TL;DR
This paper presents a method for automatically generating articulatory synthesizer control signals (gestural scores) from recorded speech and EMA data. This approach aims to create training data for articulatory TTS systems, moving towards more natural and controllable speech synthesis by learning from real human vocal tract movements.
What Problem Does This Paper Address?
Current text-to-speech (TTS) systems often rely on statistical models trained on large amounts of speech data. While these systems are good, achieving highly natural and controllable speech, especially with specific vocal tract behaviors, remains a challenge. Articulatory synthesizers, which model the physical movements of the vocal tract, offer a path to more natural speech but require detailed control signals (gestural scores) that are difficult to generate automatically. Manually creating these scores is time-consuming and impractical for large-scale TTS. This paper addresses the problem of automatically generating these gestural scores from real speech recordings, specifically using electromagnetic articulography (EMA) data, to create training data for articulatory TTS.
Methods
The core of the method involves an "analysis-by-synthesis" approach.
- Gestural Score Generation: The process starts with a known phonetic transcription of an utterance. A finite state automaton (FSA) is used to generate a vast number of possible gestural score candidates. Each candidate represents a sequence of articulatory gestures over time.
- Vocal Tract Model Synthesis: The VocalTractLab (VTL) synthesizer, a 3D geometric model of the vocal tract, is used to generate "virtual" EMA (VEMA) trajectories for each candidate gestural score.
- Error Metric and Similarity: A cost function is employed to quantify the similarity between the VEMA trajectories and the actual EMA trajectories recorded from a human speaker. This cost function combines correlation and squared error, weighted by the articulatory relevance of different vocal tract components (e.g., tongue tip, lip) for specific speech sounds.
- Articulator Relevance Weighting: To improve the accuracy of the similarity measure, the paper explores automatically identifying which articulators are most relevant for producing each phoneme. This is done using a statistical approach based on Kullback-Leibler divergence applied to the EMA data. However, preliminary results with this automatic weighting were unsatisfactory due to data limitations, leading to the use of a manually assigned baseline weighting for the current study.
- Viterbi Search: To efficiently find the best gestural score among the many possibilities, a Viterbi search algorithm is used. This dynamic programming technique navigates the "transition network" of possible gestural sequences to find the one that minimizes the cost function.
Datasets
- German EMA Corpus: A corpus of 24 normal German utterances, recorded 7 times each, from a single male speaker using electromagnetic articulography (EMA) at 200 Hz. This data was collected at the Centre for General Linguistics (ZAS), Berlin.
- Vocal Tract Model Data: The vocal tract model's geometry and target configurations are based on MRI data from the same male speaker.
Key Insights and Ideas
- Bridging the Gap: The paper demonstrates a practical method for bridging the gap between acoustic speech, physical vocal tract movements (EMA), and a controllable articulatory synthesizer (VTL).
- Unsupervised Learning Potential: By resynthesizing real speech, the system can learn gestural timings without explicit manual annotation of every gesture's timing, paving the way for more unsupervised training of articulatory TTS models.
- Articulatory Relevance is Key: The importance of weighting the similarity measure based on the relevance of specific articulators for particular sounds is highlighted. This allows the system to prioritize matching movements of critical articulators.
- Data Challenges: The study underscores the significant impact of data quality and quantity on the success of articulatory modeling and automatic feature identification. Limitations in the EMA data (e.g., coil placement, phonetic balance) directly affected the performance of the automatic relevance identification.
Key Results
- The paper successfully resynthesized a German utterance ("Ich habe Daten analysiert") using an automatically generated gestural score.
- Figure 2 shows that the resynthesized TTy (tongue tip y-axis) trajectory closely matches the original EMA trajectory, prioritizing overall shape.
- Figure 3 shows a resynthesized waveform that is qualitatively similar to the original, with an automatically aligned acoustic segmentation.
- The preliminary results of the automatic identification of relevant articulators were "somewhat unsatisfactory," attributed to data limitations rather than the algorithm itself.
Limitations and Future Work
- Vocal Tract Geometry Mismatch: Despite using the same speaker for EMA and VTL model configuration, differences in coil placement, model parametrization, and the supine vs. upright recording positions led to discrepancies, necessitating further work on vocal tract adaptation.
- Data Sparsity and Balance: The limited number of utterances and lack of phonetic balance in the initial German EMA corpus hindered the automatic identification of articulatory relevance.
- Future Work:
- Testing the automatic relevance identification on a larger, phonetically balanced English EMA corpus with more comprehensive coil coverage (including velum).
- Exploring vocal tract adaptation techniques to better align the VTL model with EMA data.
- Using the generated gestural scores to train statistical models (e.g., CART, statistical parametric models) for direct prediction of gestural timings from text or for generating parameter trajectories.
- Investigating the use of a 3D variant of the relevance identification algorithm.
Summary updated