Abstract
This paper presents a method to predict the movement of a speaker's mouth from text input using hidden Markov models (HMM). A corpus of human articulatory movements, recorded by electromagnetic articulography (EMA), was used to train the HMMs. To generate articulatory trajectories, a suitable model sequence is selected and the maximum-likelihood parameter generation (MLPG) algorithm is applied. Experiments show that fully context-dependent models outperform monophone and quinphone models, achieving an average root mean square (RMS) error of 1.945mm with predicted durations and 0.872mm with natural durations. Analysis reveals that "critical articulators"—those essential for specific sounds—are predicted with significantly higher accuracy.
Introduction
While speech technology traditionally focuses on acoustic output, modeling the physical articulation of speech is vital for applications like language tutoring, realistic 3D animation (talking heads), and articulatory speech synthesis. The primary challenge in this domain is coarticulation: the movement of articulators for a specific sound is heavily influenced by the preceding and following phonetic context. This paper adapts HMM-based parametric speech synthesis techniques to predict these complex physical trajectories directly from text.
Methodology
The researchers employed a framework similar to HMM-based speech synthesis:
- Data Acquisition: Electromagnetic Articulography (EMA) tracked the 3D coordinates of sensors on the tongue and lips.
- HMM Training: Fully context-dependent models were trained, incorporating linguistic features such as surrounding phones, lexical stress, and sentence position. This context is vital for capturing coarticulation effects.
- Trajectory Generation: The Maximum-Likelihood Parameter Generation (MLPG) algorithm was used to produce smooth, natural trajectories that respect physical constraints like velocity and acceleration, preventing erratic jumps between points.
- Duration Modeling: The system predicts the timing of sounds directly from text, which is a key requirement for autonomous text-to-movement systems.
Dataset
The study utilized the mngu0 database, featuring a single male British English speaker. It contains 1,263 phonetically balanced sentences recorded with a Carstens AG500 EMA. Twelve features were extracted, representing the x (front-to-back) and y (bottom-to-top) coordinates of six sensors: tongue dorsum (T3), tongue body (T2), tongue tip (T1), lower lip (LL), upper lip (UL), and lower incisor (LI).
The Critical Articulator Hypothesis
A central finding of this research is the "Critical Articulator Hypothesis." In linguistics, a critical articulator is the specific part of the vocal tract required to produce a given sound (e.g., the tongue tip for alveolar sounds like /t/). The study demonstrates that the HMM-based model is significantly more accurate at predicting the movement of these essential components compared to non-critical ones.
As shown in Figure 2, the Root Mean Square Error (RMSE) is consistently lowest for the articulator most involved in the sound's production. For instance, the RMSE for the tongue dorsum (T3) is minimized during the production of Velar sounds, where it acts as the critical articulator.
Results and Analysis
The best-performing model achieved an overall RMS error of 1.945 mm using predicted durations. When using natural (ground-truth) state durations, the error dropped to 0.872 mm, indicating that while spatial modeling is highly effective, temporal prediction remains a significant challenge for articulatory synthesis.
Figure 4 illustrates the normalized RMSE across various phonetic categories. For "Velar" sounds, the error for the back of the tongue (T3_y) is markedly lower than for other sensors, reinforcing the critical articulator finding. Similarly, for "Labial" sounds, the lower lip (LL_y) shows high precision compared to tongue sensors.
Conclusion
This work demonstrates that HMMs can effectively capture the complex, context-dependent nature of human articulation. The discovery that critical articulators are predicted with higher precision suggests that future models could be optimized by weighting these essential movements more heavily during training. While the current study is limited to a single speaker, it provides a robust framework for future multi-speaker articulatory modeling and more realistic speech animation.
Summary updated