2 | | |
3 | | |
4 | | = '''Voice Import Tools Tutorial : How to build a HMM-based voice for the MARY platform''' = |
5 | | |
6 | | For creating HMM-based voices we use a version of the speaker dependent training scripts provided by [http://hts.sp.nitech.ac.jp/ HTS] that was adapted to the MARY |
7 | | platform. The steps for building a HMM voice for the MARY platform can be summarised in:[[BR]] |
8 | | |
9 | | I) Checking the necessary programs and files[[BR]] |
10 | | II) Data preparation[[BR]] |
11 | | III) Training of HMM models[[BR]] |
12 | | IV) Adding a new HMM voice in the Mary system.[[BR]] |
13 | | V) Creating other voice in German or English (__if you want to train HMMs with another speech database__).[[BR]] |
14 | | |
15 | | The previous steps will be explained below creating a HMM voice using the HTS '''speaker dependent training demo'''.[[BR]] |
16 | | |
17 | | For an explanation on how to create an adapted HMM voice using the '''speaker adaptation/adaptive training demo''' please see [wiki:HMMVoiceCreationAdapt]. [[BR]] |
18 | | |
19 | | The training scripts used here are the latest versions, that is, it is required HTS_2.0.1 and SPTK-3.1. Some scripts have been added-modified to:[[BR]] |
| 2 | = '''Voice Import Tools Tutorial : How to build a HMM-based voice for the MARY 4.1.0 platform''' = |
| 3 | |
| 4 | For creating HMM-based voices we use a version of the speaker dependent training scripts provided by [http://hts.sp.nitech.ac.jp/ HTS] that was adapted to the MARY 4.1.0 platform. The steps for building a HMM voice for the MARY platform can be summarised in:[[BR]] |
| 5 | |
| 6 | I) Download MARY TTS including Voice import tools[[BR]] |
| 7 | II) Check necessary programs and files[[BR]] |
| 8 | III) Check data: audio and text files[[BR]] |
| 9 | IV) Run the Voice import tools [[BR]] |
| 10 | V) Creating other voice in a language different from German or English (US). |
| 11 | |
| 12 | The training scripts used here are the latest versions, that is, it is required HTS_2.1 and SPTK-3.2. Some scripts have been added-modified to:[[BR]] |
21 | | - Train bandpass voicing strengths for mixed excitation.[[BR]] |
22 | | - Process language specific settings as parameters.[[BR]] |
23 | | |
24 | | |
25 | | ''' |
26 | | === I) Checking the necessary programs and files: === |
27 | | ''' |
28 | | |
29 | | MARY requirements:[[BR]] |
30 | | - Operating System - Linux[[BR]] |
31 | | - MARY TTS Recent Version - Download Link: http://mary.dfki.de/Download [[BR]] |
32 | | - Openmary - SVN from http://mary.opendfki.de [[BR]] |
33 | | - MARY patch for HTS demo: HTS-2.0.1-demo_CMU-ARCTIC-SLT_for_Mary-3.6.0.patch [[BR]] |
34 | | |
35 | | |
36 | | HTS requirements, please download and follow the instructions for installing:[[BR]] |
37 | | - HTS-2.0.1_for_HTK-3.4.patch [[BR]] |
38 | | - HTK-3.4 ( [wiki:CompilingHTK Important note on compiling HTK] ) [[BR]] |
39 | | - SPTK-3.1 [[BR]] |
40 | | - HTS-demo_CMU-ARCTIC-SLT (for HTS-2.0.1) [[BR]] |
41 | | |
42 | | Other requirements, please download and follow the instructions for installing: [[BR]] |
43 | | - EHMM for automatic labeling, available with festvox-2.1 (Recent Version) http://festvox.org/download.html [[BR]] |
44 | | - sox, normally available in linux. [[BR]] |
45 | | - tcl-tk supporting snack, for example ActiveTcl - Download Link: http://www.activestate.com/Products/ActiveTcl/ [[BR]] |
46 | | - perl, normally available in linux. [[BR]] |
47 | | |
48 | | 0.1) download and un-zip, un-tar the latest speaker dependent training demo for English. |
49 | | |
50 | | http://hts.sp.nitech.ac.jp/archives/2.0.1/HTS-demo_CMU-ARCTIC-SLT.tar.bz2 for HTS-2.0.1 |
51 | | |
52 | | 0.2) download and unzip the patch file for using MARY instead of Festival as text analyser. |
53 | | |
54 | | https://mary.opendfki.de/repos/trunk/lib/hts/HTS-2.0.1-demo_CMU-ARCTIC-SLT_for_Mary-3.6.0.patch.zip [[BR]] |
55 | | |
56 | | apply the patch to the HTS-demo_CMU-ARCTIC-SLT directory: [[BR]] |
57 | | {{{ |
58 | | patch -p1 -d . < HTS-2.0.1-demo_CMU-ARCTIC-SLT_for_Mary-3.6.0.patch |
59 | | }}} |
60 | | |
61 | | 0.3) create a wav directory. |
62 | | |
63 | | 0.4) Run the VoiceImport program |
64 | | |
65 | | First of all you need to set your MARY_BASE directory and then run the program: [[BR]] |
66 | | {{{ |
67 | | export MARY_BASE="/dir/to/openmary" |
68 | | java -jar -Xmx1024m $MARY_BASE/java/voiceimport.jar |
69 | | }}} |
70 | | |
71 | | If you are not familiar or have problems with the VoiceImport program, please read and follow the instructions in the Voice Import Tools |
72 | | Tutorial: http://mary.opendfki.de/wiki/VoiceImportToolsTutorial |
73 | | |
74 | | If you want to create another voice in German or English please see the section V below. |
75 | | |
76 | | Please remember that whenever you are in doubt about the settings of a particular component you can check its corresponding help for a description of the meaning |
77 | | (and possible values) of each variable. |
78 | | |
79 | | ''' |
80 | | === II) Data preparation: === |
81 | | ''' |
82 | | |
83 | | 1- Run the HMMVoiceDataPreparation of the HMM Voice Trainer group to check if text, wav and data/raw files are available and in the correct paths. |
84 | | If just data/raw is provided, the program will do the conversion. If no text files are available but data/utts in festival format, the program will do the |
85 | | conversion as well. |
86 | | |
87 | | 2- Run the PhoneUnitFeatureComputer component of the Feature Extraction group to extract context feature vectors from the text data. This procedure will create a "phonefeatures" directory. For running this component the MARY server should be running as well. |
88 | | |
89 | | 3- Run the EHMMlabeler component of the Automatic Labeling group to label automatically the wav files using the corresponding transcriptions. This procedure might |
| 14 | - Train bandpass voicing strengths and Fourier magnitudes for mixed excitation.[[BR]] |
| 15 | |
| 16 | '''MARY requirements:'''[[BR]] |
| 17 | - Operating System - Linux (tested on Ubuntu 9.04) [[BR]] |
| 18 | - MARY TTS 4.1.0 including Voice import tools during installation - link: [http://mary.dfki.de/download/4.0/openmary-standalone-install-4.0.0.jar MARY TTS 4.1.0] [[BR]] |
| 19 | |
| 20 | |
| 21 | ''' |
| 22 | == I) Download MARY TTS including Voice import tools == |
| 23 | ''' |
| 24 | |
| 25 | Click on the latest MARY release [http://mary.dfki.de/download/4.0/openmary-standalone-install-4.0.0.jar MARY download] or download the file and run it with: |
| 26 | {{{ |
| 27 | java -jar openmary-standalone-install-4.1.0.jar |
| 28 | }}} |
| 29 | |
| 30 | |
| 31 | ''' |
| 32 | == II) Check the necessary programs and files: == |
| 33 | ''' |
| 34 | |
| 35 | We provide an script to facilitate the checking and installation of the necessary external programs, once installed MARY TTS open a command line shell in your voice building directory and run the shell script: |
| 36 | {{{ |
| 37 | $MARY_BASE/lib/external/check_install_external_programs.sh |
| 38 | }}} |
| 39 | |
| 40 | With the option '''-check''', this script will check if the necessary programs and versions are installed (that is, the programs can be found in the PATH or in the paths provided by the user).[[BR]] |
| 41 | With the option '''-install''' this script will try to download and install the necessary programs in: $MARY_BASE/lib/external/bin (if problems, it will suggest how to install manually the programs). |
| 42 | |
| 43 | If you have already installed some of the required programs, '''please include their paths in the PATH variable or provide the paths''', for example: |
| 44 | {{{ |
| 45 | $MARY_BASE/lib/external/check_install_external_programs.sh -check /your/path/to/htk/bin /your/path/to/Festival/festvox/src/ehmm/bin |
| 46 | }}} |
| 47 | |
| 48 | This script generates a '''$MARY_BASE/lib/external/externalBinaries.config''' file that will be used by the Voice import tools to locate the necessary external programs. |
| 49 | |
| 50 | The necessary programs that this script checks are:[[BR]] |
| 51 | |
| 52 | '''HTS requirements:'''[[BR]] |
| 53 | - [http://hts.sp.nitech.ac.jp/archives/2.1/HTS-2.1_for_HTK-3.4.tar.bz2 HTS-2.1_for_HTK-3.4.patch] [[BR]] |
| 54 | - HTK-3.4 and HDecode patched with HTS-2.1_for_HTK-3.4.patch links: |
| 55 | * [http://htk.eng.cam.ac.uk/ftp/software/HTK-3.4.tar.gz HTK-3.4] (you will need to register first) [[BR]] |
| 56 | * [http://htk.eng.cam.ac.uk/prot-docs/hdecode.shtml HDecode] (you will need to register first) [[BR]] |
| 57 | - [http://downloads.sourceforge.net/sp-tk/SPTK-3.2.tar.gz SPTK-3.2] [[BR]] |
| 58 | - [http://downloads.sourceforge.net/hts-engine/hts_engine_API-1.01.tar.gz hts_engine_API-1.01] [[BR]] |
| 59 | |
| 60 | '''Other requirements:'''[[BR]] |
| 61 | - awk normally available in linux [[BR]] |
| 62 | - perl normally available in linux [[BR]] |
| 63 | - bc normally available in linux [[BR]] |
| 64 | - sox, v13.0 or greater [http://sox.sourceforge.net/ SoX], normally available in linux. [[BR]] |
| 65 | - tcl supporting snack, for example [http://www.activestate.com/Products/ActiveTcl/ ActiveTcl.] Note that only ActiveTcl 8.4 includes snack; 8.5+ requires manual installation. [[BR]] |
| 66 | - [http://www.speech.kth.se/snack/download.html snack] library for tcl. [[BR]] |
| 67 | - EHMM for automatic labeling, available with [http://festvox.org/download.html festvox-2.1] [[BR]] |
| 68 | |
| 69 | |
| 70 | |
| 71 | |
| 72 | ''' |
| 73 | == III) Check data: audio and text files[[BR]] == |
| 74 | ''' |
| 75 | |
| 76 | In your voice building directory execute the step-by-step procedure in [http://mary.opendfki.de/wiki/VoiceImportToolsTutorial VoiceImportToolsTutorial] to make |
| 77 | sure that the data, sound (wav) and text files are in the correct place and format.[[BR]] |
| 78 | |
| 79 | As a result of this step your voice building directory should contain a wav and text directories. |
| 80 | |
| 81 | |
| 82 | ''' |
| 83 | == IV) Run the Voice Import tools == |
| 84 | ''' |
| 85 | |
| 86 | In your voice building directory run the voice import tools: |
| 87 | {{{ |
| 88 | export MARY_BASE="/your/path/to/MARY TTS/" |
| 89 | java -Xmx1024m -jar $MARY_BASE/java/voiceimport.jar |
| 90 | }}} |
| 91 | |
| 92 | After starting the Voice Import Tools check the global settings of the voice, make sure that the allophones file is provided and exists: |
| 93 | {{{ |
| 94 | db.alophonesSet = $MARY_BASE/lib/modules/xx/lexicon/allophones.xx.xml (where xx is the corresponding language) |
| 95 | }}} |
| 96 | |
| 97 | |
| 98 | And run the following components: |
| 99 | |
| 100 | |
| 101 | '''1-''' Run the HMMVoiceDataPreparation of the HMM Voice Trainer group to set up the environment to create a HMM voice and check if required external programs and text and wav files are available and in the correct paths. |
| 102 | |
| 103 | '''2-''' Run the AllophonesExtractor of the Automatic Labeling group to create the '''prompt_allophones''' directory required in the next step. This component requires the MARY server. [[BR]] |
| 104 | |
| 105 | '''3-''' Run the EHMMlabeler component of the Automatic Labeling group to label automatically the wav files using the corresponding transcriptions. This procedure might |
99 | | |
100 | | 5- Run the PhoneUnitLabelComputer component of the Labels and Pause Correction group. This procedure will create a "phonelab" directory. |
101 | | |
102 | | 6- Run the PhonelabelFeatureAligner component of the Labels and Pause Correction group. This procedure will verify alignment between "phonefeatures" and "phonelabels". |
103 | | |
104 | | |
105 | | ''' |
106 | | === III) HMM models training: === |
107 | | ''' |
108 | | |
109 | | 7- Run the HMMVoiceConfigure component of the HMM Voice trainer group, the default setting values of this component are already fixed for the |
110 | | HTS-demo_CMU-ARCTIC-SLT voice. |
| 116 | The result of this step is a '''lab''' directory. |
| 117 | |
| 118 | '''5-''' Run the TranscriptionAligner component of the Label-Transcript Alignment group. This program will create the '''allophones''' directory. |
| 119 | |
| 120 | '''6-''' Run the PhoneUnitLabelComputer component of the Label-Transcript Alignment group. This procedure has as input the '''lab''' directory and will create as an output the '''phonelab''' directory. |
| 121 | |
| 122 | '''7-''' Run the FeatureSelelection component of the Feature Extraction group. This program will create a '''mary/features.txt''' file, it requires the MARY server running. Select here all the features and save the file. |
| 123 | |
| 124 | '''8-''' Run the PhoneUnitFeatureComputer component of the Feature Extraction group to extract context feature vectors from the text data. This procedure will create a '''phonefeatures''' directory. For running this component the MARY server should be running as well. |
| 125 | |
| 126 | '''9-''' Run the PhonelabelFeatureAligner component of the Verify Alignment group. This procedure will verify alignment between "phonefeatures" and "phonelabels".[[BR]] |
| 127 | |
| 128 | As a result of previous steps we should have:[[BR]] |
| 129 | - phonefeatures directory [[BR]] |
| 130 | - phonelab directory [[BR]] |
| 131 | - mary/features.txt file [[BR]] |
| 132 | - $MARY_BASE/lib/external/externalBinaries.config |
| 133 | |
| 134 | |
| 135 | ''' |
| 136 | === HMM models training: === |
| 137 | ''' |
| 138 | |
| 139 | '''10-''' Run the HMMVoiceConfigure component of the HMM Voice trainer group. The default setting values are already fixed for the arctic slt voice, some path settings depend on your installation, and will be taken from $MARY_BASE/lib/external/externalBinaries.config |
119 | | HMMVoiceConfigure.voiceLang = de |
120 | | }}} |
121 | | |
122 | | Using the settings editor of this component you can also change other variables like using LSP instead og MGC, sampling frequency, etc., |
123 | | the same as you would do when running "make configure + parameters" with the original HTS scripts. |
124 | | |
125 | | 8- Run the HMMVoiceMakeData component of the HMM Voice trainer group to run the HTS procedure "make data". This procedure is the same as in the original scripts with additional sections for calculating strengths (for mixed excitation), global variance, and handling of MARY context features. |
126 | | |
127 | | Particular procedures can be repeated isolated fixing the particular settings for this component. For example, if the procedure that creates strengths (in the str directory) has to be repeated with a different set of filters (data/filters/), please remove the old str directory and set: |
| 147 | }}} |
| 148 | |
| 149 | Using the settings editor of this component you can also change other variables like using LSP instead og MGC, sampling frequency, etc., the same as you would do when running "make configure + parameters" with the original HTS scripts. |
| 150 | |
| 151 | '''11-''' Run the HMMVoiceFeatureSelection component of the HMM Voice trainer group. This program reads the '''mary/features.txt''' file (created in step 11), and generates the file '''mary/hmmFeatures.txt'''. This file contains extra features, apart from phone and phonological features, that will be used to train HMMs. When running this program a small set of features will be presented on top, separated by an empty line:[[BR]] |
| 152 | {{{ |
| 153 | pos_in_syl |
| 154 | syl_break |
| 155 | prev_syl_break |
| 156 | position_type |
| 157 | |
| 158 | accented |
| 159 | accented_syls_from_phrase_end |
| 160 | accented_syls_from_phrase_start |
| 161 | breakindex |
| 162 | edge |
| 163 | ... |
| 164 | }}} |
| 165 | If you are not sure about using other features, use the first four, delete the others and save the file. |
| 166 | |
| 167 | '''12-''' Run the HMMVoiceMakeData component of the HMM Voice trainer group to run the HTS procedure "make data". This procedure require the following files: |
| 168 | {{{ |
| 169 | HMMVoiceMakeData.allophonesFile = allophones.en_US.xml # allophones set (language dependent) |
| 170 | HMMVoiceMakeData.featureListFile = mary/hmmFeatures.txt # extra context features used for training HMMs. |
| 171 | }}} |
| 172 | |
| 173 | The allophones set file is language dependent, it can be found in $MARY_BASE/lib/modules/en/us/lexicon/allophones.en_US.xml[[BR]] |
| 174 | The hmmFeatures.txt is the file created in step 15 and contains additional context features, apart from phone and phonological features, used for training HMMs.[[BR]] |
| 175 | |
| 176 | The HMMVoiceMakeData procedure is similar to the original HTS scripts with additional sections for calculating strengths, Fourier magnitudes (for mixed excitation), global variance and composing training data files from mgc, lf0, str and mag files. This component will execute in the hts/data/ directory: |
| 177 | {{{ |
| 178 | make all-mary or |
| 179 | make mgc lf0 str-mary mag-mary cmp-mary gv-mary gv list scp |
| 180 | }}} |
| 181 | The '''label''' directory and the '''mlf''' files in MARY are done with the Voice Import Tools: HMMVoiceMakeData.makeLabels()[[BR]] |
| 182 | The '''questions''' file in MARY is done with the Voice Import Tools: HMMVoiceMakeData.makeQuestions() |
| 183 | |
| 184 | |
| 185 | Particular procedures can be repeated isolated fixing the particular settings for this component. For example, if the procedure that creates strengths (in the str directory) has to be repeated with a different set of filters (data/filters/), set: |
163 | | }}} |
164 | | |
165 | | The VoiceInstaller will: [[BR]] |
166 | | - Create a new mary config file in: $MARY_BASE/conf/german-hmm-voice.config [[BR]] |
167 | | - Add the files corresponding to this voice in: $MARY_BASE/lib/voices/hmm-voice/ [[BR]] |
168 | | - copy features list: data/feature_list_en.pl to $MARY_BASE/lib/voices/hmm-voice [[BR]] |
169 | | - copy one example of phonelab for testing the synthesiser: data/labels/gen/gen_cmu_us_arctic_slt_xxxx.lab to $MARY_BASE/lib/voices/hmm-voice [[BR]] |
170 | | - copy the HTS trees: voices/qst001/ver1/*.inf to $MARY_BASE/lib/voices/hmm-voice [[BR]] |
171 | | - copy the HTS PDF models: voices/qst001/ver1/*.pdf to $MARY_BASE/lib/voices/hmm-voice [[BR]] |
172 | | - copy global variance models (if useGV is set to true): data/gv/gv-*-littend.pdf to $MARY_BASE/lib/voices/hmm-voice [[BR]] |
173 | | - copy filter taps for mixed excitation: data/filters/mix_excitation_filters.txt to $MARY_BASE/lib/voices/hmm-voice [[BR]] |
174 | | |
175 | | After successfully installing a new voice, it can be used with the mary_server and the mary_client. |
176 | | |
177 | | |
178 | | ''' |
179 | | === V) Creating other voice in German or English. === |
180 | | ''' |
181 | | |
182 | | If using German: |
183 | | |
184 | | For creating a new German voice it is necessary: [[BR]] |
185 | | * a wav or raw directory with the speech files you will use for training the German voice. [[BR]] |
186 | | * transcriptions of the files, one text file per speech file, or transcriptions in festival format if available. [[BR]] |
187 | | |
188 | | Then we use as a base the original HTS-demo_CMU-ARCTIC-SLT directory: |
189 | | |
190 | | - Download and un-zip, un-tar the HTS-demo_CMU-ARCTIC-SLT for HTS-2.0.1 |
191 | | |
192 | | - Rename this directory as your new voice name, for example german_voice, and delete the directories data/raw and data/utt. |
193 | | |
194 | | - Apply the MARY patch to the german_voice directory. [[BR]] |
195 | | patch -p1 -d . < HTS-2.0.1-demo_CMU-ARCTIC-SLT_for_Mary-3.5.0.patch |
196 | | |
197 | | - Move your speech files to this directory, if you have a wav directory, this should be copied in the current directory (german_voice/wav). If you have a raw directory, this should be copied in the data directory (german_voice/data/raw). |
198 | | |
199 | | - Move your transcription files to this directory, if you have a text directory containing the transcription of each file in separate files, this should be copied in the current directory (german_voice/text). If you have transcriptions in festival format please copy this directory in the data/utts directory (german_voice/data/utts/). |
200 | | |
201 | | - Now run the VoiceImport program and follow the instructions as normal. Provide general settings for: |
| 218 | HMMVoicePackager.useAcousticModels = true |
| 219 | set this variable to true to allow prosody modification specified in MARYXML |
| 220 | }}} |
| 221 | |
| 222 | The HMMVoicePackager will pack in a zip file located in MARY_BASE/download the following files: [[BR]] |
| 223 | - A mary config file: german-hsmm-voice.config [[BR]] |
| 224 | - HMM files corresponding to this voice: [[BR]] |
| 225 | - one example of phonefeatures for testing the synthesiser: data/phonefeatures/cmu_us_arctic_slt_xxxx.pfeats [[BR]] |
| 226 | - the HTS trees: voices/qst001/ver1/*.inf [[BR]] |
| 227 | - the HTS PDF models: voices/qst001/ver1/*.pdf [[BR]] |
| 228 | - global variance models (if useGV is set to true): data/gv/gv-*-littend.pdf [[BR]] |
| 229 | - filter taps for mixed excitation: data/filters/mix_excitation_filters.txt [[BR]] |
| 230 | - trickyPhones.txt file, if one was created during training [[BR]] |
| 231 | |
| 232 | After successfully packing a new voice, you must run the MARY Component Installer to install the voice! |
| 233 | |
| 234 | |
| 235 | ''' |
| 236 | === V) Creating other voice in a language different from German or English (US). === |
| 237 | ''' |
| 238 | |
| 239 | If you are creating a voice in other language you will need to specify: |
| 240 | |
| 241 | - '''Minimal NLP components''': if you are creating a new voice from scratch, for example following the steps in [http://mary.opendfki.de/wiki/NewLanguageSupport NewLanguageSupport], you will need to create Minimal NLP components for the new language. These minimal components are necessary to run the MARY server in the new language and extract context features ('''phonefeatures''' directory). |
| 242 | |
| 243 | - '''Phoneme set''': contained in $MARY_BASE/lib/modules/xx/lexicon/allophones.xx.xml , where xx corresponds to the new language. |
| 244 | |
| 245 | - After creating the minimal components, you will need wav files (in a wav directory) and the corresponding transcriptions (one file per wav file in a text directory). [[BR]] |
| 246 | Afterwards follow the instructions as normal from step 1. Provide general settings for: |