Ultrax: an animated midsagittal vocal tract display for speech therapy

Korin Richmond, Steve RenalsInterspeech 2012

TL;DR

This paper introduces "Ultrax," a system that creates real-time animated visualizations of the vocal tract for speech therapy. By combining MRI-based vocal tract shape models with EMA-driven animation, Ultrax aims to provide a more intuitive visual aid for diagnosing and treating speech sound disorders, moving beyond traditional auditory-based therapy.

What Problem Does This Paper Address?

Speech sound disorders (SSDs) are common childhood communication impairments that can significantly impact social development and learning. Current speech therapy for SSDs heavily relies on auditory feedback, which can be challenging for children with impaired auditory skills. While some visual aids exist (like mirrors for lip articulation or electropalatography for tongue-palate contact), there's a lack of real-time visual feedback for internal articulation within the oral cavity. The Ultrax project aims to fill this gap by providing an animated, diagrammatic display of the vocal tract, making articulation more understandable and aiding both diagnosis and therapy.

Methods

The core of the Ultrax approach involves two main components:

  1. Dimensionality-Reducing Vocal Tract Shape Model: This component maps a small set of control parameters to a representation of the vocal tract's shape. The authors initially implemented a linear Principal Component Analysis (PCA) model based on midsagittal vocal tract shapes derived from MRI scans. This allows for a simplified representation of complex vocal tract configurations.
  2. Latent Space Model: This model captures the temporal dynamics of the vocal tract. It uses hidden variables to represent the state of the vocal tract over time. The system infers these hidden states from observable articulatory data (initially EMA) and uses them to drive the dimensionality-reducing shape model. The authors employed a standard Kalman filter for state estimation, suitable for their linear model.

The process involves:

  • Contour Extraction: Hand-labeling vocal tract structures (lips, palate, tongue, pharynx) from MRI scans.
  • Normalization: Resampling contours to a fixed number of points and correcting for head pose variations, primarily using the hard palate as a reference.
  • EMA-Driven Animation: Aligning EMA coil data (representing fleshpoint positions) with the MRI coordinate system and the derived vocal tract shapes. This alignment is crucial for driving the animation with real-time articulatory measurements.

Datasets

  • mngu0 corpus: This corpus was used for both training and evaluation. It contains:
  • 3D MRI scans of 28 phones.
  • Approximately 1300 utterances recorded using Electromagnetic Articulography (EMA) from the same subject. EMA coils were placed on the lips, lower incisor, and three points on the tongue.

Key Insights and Ideas

  • Bridging Modalities: The paper demonstrates a novel way to bridge data from different imaging modalities (MRI for static shape models, EMA for dynamic articulation) to create a unified, animated vocal tract display.
  • Simplification for Therapy: Recognizing that raw ultrasound data is difficult to interpret, the Ultrax project focuses on creating a simplified, diagrammatic display. This is key for making the visualization useful in a clinical setting.
  • Latent Space for Dynamics: The use of a latent space model is crucial for capturing the smooth, continuous movements of the vocal tract over time, rather than just static snapshots.
  • PCA for Shape Representation: PCA proves effective in reducing the dimensionality of vocal tract shapes, allowing for animation with a manageable number of control parameters.
  • EMA as a Proof-of-Concept: While the ultimate goal is ultrasound, using EMA data for the initial prototype simplifies the problem by providing a more direct articulatory measurement, allowing the core animation framework to be tested.

Key Results

  • Successful Animation: The system successfully animated a vocal tract model driven by EMA data.
  • Evaluation Metrics:
    • Tracking EMA Coils: The Kalman filter successfully tracked EMA coil positions. The root mean square (RMS) error for tracking EMA coils was reduced to approximately zero when the hidden state size (number of PCA components) reached twelve.
  • Leave-One-Out Prediction: When predicting the position of one EMA coil while leaving it out of the observation vector, the RMS error was generally slightly higher. The optimal hidden state dimensionality for this task was found to be around four for salient articulators like the front tongue points.
  • Comparison to Linear Mappings: The proposed approach (using four hidden state dimensions) showed an average prediction error of 2.9mm, which was only slightly worse (0.4mm) than a more favorable direct linear mapping (2.5mm) and significantly better than a lower bound error (1.1mm).

Limitations and Future Work

  • Manual Alignment: The alignment of EMA data to MRI coordinates and the labeling of corresponding points on VT contours were performed "by eye." This manual process is a potential source of error.
  • EMA Coil Limitations: The exact fleshpoint location corresponding to an EMA coil is not precisely known due to coil construction, introducing uncertainty.
  • Speaker-Dependent: The current model is speaker-dependent, trained on data from a single subject.
  • Future Work: The authors explicitly state their ultimate goal is to use ultrasound data for real-time animation. They also mention the need to explore more sophisticated latent space models (e.g., particle filters) if nonlinear F or H functions are used. The paper suggests that the optimal hidden state dimensionality might be lower than expected, indicating that limited EMA observations may not fully constrain models with many degrees of freedom.

Summary updated

BibTeX
@inproceedings{richmond2012ultrax,
  author = {Korin Richmond and Steve Renals},
  title = {{Ultrax: an animated midsagittal vocal tract display for speech therapy}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure