HMM-based text-to-articulatory-movement prediction and analysis of critical articulators

Zhen-Hua Ling, Korin Richmond, Junichi YamagishiInterspeech 2010 · OpenReview

Abstract

This paper presents a method to predict the movement of a speaker's mouth from text input using hidden Markov models (HMM). A corpus of human articulatory movements, recorded by electromagnetic articulography (EMA), was used to train the HMMs. To generate articulatory trajectories, a suitable model sequence is selected and the maximum-likelihood parameter generation (MLPG) algorithm is applied. Experiments show that fully context-dependent models outperform monophone and quinphone models, achieving an average root mean square (RMS) error of 1.945mm with predicted durations and 0.872mm with natural durations. Analysis reveals that "critical articulators"—those essential for specific sounds—are predicted with significantly higher accuracy.

Introduction

While speech technology traditionally focuses on acoustic output, modeling the physical articulation of speech is vital for applications like language tutoring, realistic 3D animation (talking heads), and articulatory speech synthesis. The primary challenge in this domain is coarticulation: the movement of articulators for a specific sound is heavily influenced by the preceding and following phonetic context. This paper adapts HMM-based parametric speech synthesis techniques to predict these complex physical trajectories directly from text.

Methodology

The researchers employed a framework similar to HMM-based speech synthesis:

  1. Data Acquisition: Electromagnetic Articulography (EMA) tracked the 3D coordinates of sensors on the tongue and lips.
  2. HMM Training: Fully context-dependent models were trained, incorporating linguistic features such as surrounding phones, lexical stress, and sentence position. This context is vital for capturing coarticulation effects.
  3. Trajectory Generation: The Maximum-Likelihood Parameter Generation (MLPG) algorithm was used to produce smooth, natural trajectories that respect physical constraints like velocity and acceleration, preventing erratic jumps between points.
  4. Duration Modeling: The system predicts the timing of sounds directly from text, which is a key requirement for autonomous text-to-movement systems.

Dataset

The study utilized the mngu0 database, featuring a single male British English speaker. It contains 1,263 phonetically balanced sentences recorded with a Carstens AG500 EMA. Twelve features were extracted, representing the x (front-to-back) and y (bottom-to-top) coordinates of six sensors: tongue dorsum (T3), tongue body (T2), tongue tip (T1), lower lip (LL), upper lip (UL), and lower incisor (LI).

The Critical Articulator Hypothesis

A central finding of this research is the "Critical Articulator Hypothesis." In linguistics, a critical articulator is the specific part of the vocal tract required to produce a given sound (e.g., the tongue tip for alveolar sounds like /t/). The study demonstrates that the HMM-based model is significantly more accurate at predicting the movement of these essential components compared to non-critical ones.

Sensor locations and RMSE for critical articulators across different places of articulation.

As shown in Figure 2, the Root Mean Square Error (RMSE) is consistently lowest for the articulator most involved in the sound's production. For instance, the RMSE for the tongue dorsum (T3) is minimized during the production of Velar sounds, where it acts as the critical articulator.

Results and Analysis

The best-performing model achieved an overall RMS error of 1.945 mm using predicted durations. When using natural (ground-truth) state durations, the error dropped to 0.872 mm, indicating that while spatial modeling is highly effective, temporal prediction remains a significant challenge for articulatory synthesis.

Normalized EMA RMSE across different phone types for various articulators.

Figure 4 illustrates the normalized RMSE across various phonetic categories. For "Velar" sounds, the error for the back of the tongue (T3_y) is markedly lower than for other sensors, reinforcing the critical articulator finding. Similarly, for "Labial" sounds, the lower lip (LL_y) shows high precision compared to tongue sensors.

Conclusion

This work demonstrates that HMMs can effectively capture the complex, context-dependent nature of human articulation. The discovery that critical articulators are predicted with higher precision suggests that future models could be optimized by weighting these essential movements more heavily during training. While the current study is limited to a single speaker, it provides a robust framework for future multi-speaker articulatory modeling and more realistic speech animation.

Summary updated

Abstract

In this paper we present a method to predict the movement of a speaker's mouth from text input using hidden Markov models (HMM). We have used a corpus of human articulatory movements, recorded by electromagnetic articulography (EMA), to train HMMs. To predict articulatory movements from text, a suitable model sequence is selected and the maximum-likelihood parameter generation (MLPG) algorithm is used to generate output articulatory trajectories. In our experiments, we find that fully context-dependent models outperform monophone and quinphone models, achieving an average root mean square (RMS) error of 1.945mm when state durations are predicted from text, and 0.872mm when natural state durations are used. Finally, we go on to analyze the prediction error for different EMA dimensions and phone types. We find a clear pattern emerges that the movements of so-called critical articulators can be predicted more accurately than the average performance.
BibTeX
@inproceedings{ling2010hmm,
  author = {Zhen-Hua Ling and Korin Richmond and Junichi Yamagishi},
  title = {{HMM-based text-to-articulatory-movement prediction and analysis of critical articulators}},
  booktitle = {Interspeech},
  year = {2010},
  url = {https://openreview.net/forum?id=sjDHgs0DRo},
}
Figure