Acoustic-to-articulatory inversion using speech recognition and trajectory formation based on phoneme hidden Markov models

Atef Ben Youssef, Pierre Badin, Gérard Bailly, Panikos HeracleousInterspeech 2009

TL;DR

This paper presents a novel method for "speech inversion," reconstructing the movements of hidden speech articulators (like the tongue) from audio alone. By combining speech recognition with a trajectory formation model based on Hidden Markov Models (HMMs), they achieve promising results in synthesizing articulatory movements, opening doors for richer augmented speech applications.

What Problem Does This Paper Address?

Understanding how we produce speech is a long-standing challenge. While we can easily observe lip movements, the actions of internal articulators like the tongue and velum are hidden. This paper tackles the problem of "speech inversion," aiming to reconstruct these hidden articulatory movements solely from the acoustic speech signal. This capability is crucial for applications like "augmented speech," where synthesized articulatory movements could be visualized on a virtual avatar, enhancing communication for hearing-impaired individuals or providing deeper insights into speech production. Previous methods often relied on analysis-by-synthesis or complex statistical models, but this work explores a data-driven approach using HMMs trained on synchronized acoustic and articulatory data.

Methods

The core of this work lies in a two-step inversion process:

  1. Phoneme Recognition: First, they use acoustic HMMs to recognize the sequence of phonemes and their durations from the input speech signal. This step leverages standard speech recognition techniques.
  2. Articulatory Trajectory Formation: Second, the recognized phoneme sequence and durations are fed into the articulatory part of the HMMs. This generates a sequence of articulatory movements, essentially synthesizing the trajectory of the articulators. This synthesis is performed using a parameter generation algorithm from the HTS (HMM-based Speech Synthesis) toolkit.

A key methodological aspect is the use of a multistream HMM framework. This allows them to train acoustic and articulatory HMMs simultaneously, ensuring they share the same phoneme and state boundaries. They also explore different context-dependent HMMs (no context, left context, right context, and both) to capture coarticulatory effects, which are crucial for accurate speech production modeling.

Datasets

The study utilizes a custom corpus recorded from a single male French speaker. This corpus consists of:

  • 2 repetitions of 224 nonsense VCV sequences.
  • 2 repetitions of 109 CVC real French words.
  • 68 short French sentences.
  • 9 longer phonetically balanced French sentences.
  • 11 long arbitrary sentences.

In total, the corpus contains approximately 12 minutes of speech, corresponding to 7352 phonemes. The articulatory data was collected using an ElectroMagnetic Articulograph (EMA) tracking six coils (jaw, tongue tip, tongue middle, tongue back, upper lip, lower lip), synchronized with the acoustic signal.

Key Insights and Ideas

  • The Power of Context: The results strongly suggest that incorporating phonetic context (left and right neighbors of a phoneme) significantly improves the accuracy of articulatory reconstruction. This highlights the interconnected nature of speech sounds.
  • Two-Stage Approach Effectiveness: The combination of a robust phoneme recognition step followed by a trajectory formation model proves effective. This modular approach allows for independent optimization of each stage.
  • Trajectory Formation Contributes Significantly to Error: While phoneme recognition is important, the study indicates that a substantial portion of the overall inversion error stems from the trajectory formation step itself. This points to areas for future improvement.
  • Promising Upper Bound: The "close test" scenario (where training and testing sets are identical) yields a very low RMS error (1.1 mm), suggesting the underlying model has strong potential when fully trained.

Key Results

  • Phoneme Recognition: The best phoneme recognition rate achieved was 97.9% (accuracy) when using context-dependent HMMs and the entire corpus for training. In a more realistic scenario (2/3 training, 1/3 testing), the accuracy was 77.2% for L-ctx-R, which improved to 84.8% after replacing missing HMMs.
  • Articulatory Reconstruction (RMS Error):
    • In the ideal "close test" scenario, the RMS error ranged from 1.05 mm (L-ctx-R) to 2.26 mm (no-ctx).
  • In the more realistic 2/3 training, 1/3 testing scenario, the RMS error was around 2.3 mm.
  • The authors report that the trajectory formation step contributes significantly to the overall error, with synthesized trajectories directly from original labels showing RMS errors between 1.11 mm and 1.74 mm.

Limitations and Future Work

  • Corpus Size and Speaker Dependence: The study acknowledges the relatively small size of the corpus and the fact that it was recorded from a single male speaker. This limits the generalizability of the results. Future work will involve recording a larger corpus to increase the representativity of the HMMs, especially for context-dependent models.
  • Improving Trajectory Formation: The authors identify the trajectory formation step as a significant source of error. They plan to explore using recognized state durations instead of z-scoring and optimizing the training criterion based on reconstruction error.
  • Phasing Models: Learning state-level phasing models between acoustics and articulatory boundaries is another avenue for future research.
  • Incorporating Visual Information: The paper suggests that adding visual information (e.g., from Active Appearance Models) could further enhance the inversion system.
  • Model Complexity: Exploring more complex HMM structures, such as increasing the number of Gaussians per state or using tied states and mixtures, is also planned.

Summary updated

BibTeX
@inproceedings{youssef2009acoustic,
  author = {Atef Ben Youssef and Pierre Badin and Gérard Bailly and Panikos Heracleous},
  title = {{Acoustic-to-articulatory inversion using speech recognition and trajectory formation based on phoneme hidden Markov models}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure