Statistical multi-stream modeling of real-time MRI articulatory speech data

Erik Bresch, Athanasios Katsamanis, Louis Goldstein, Shrikanth S. NarayananInterspeech 2010 · DOI

Introduction

Real-time Magnetic Resonance Imaging (rt-MRI) provides a high-fidelity window into the dynamic movements of the human vocal tract during speech. However, the complexity of these movements presents a significant analytical challenge known as spatio-temporal mixing. Articulators do not move in isolation; for instance, the mechanical action of the jaw simultaneously influences both the lips and the tongue. This "coupling" makes it difficult to isolate the underlying phonological "gestures"—the goal-directed actions of specific articulators—from passive movements.

Illustration of spatial coupling between Lip Aperture (LA) and Tongue Tip Constriction Degree (TTCD) during the production of "pay novas".

Methodology

To address these challenges, researchers at USC developed a multi-stage statistical pipeline designed to automatically segment speech into distinct articulatory states.

Feature Extraction

Rather than employing computationally expensive and error-prone boundary tracking, the researchers utilized a robust intensity-based method. Three Rectangular Regions of Interest (ROIs) were defined: the lips (Lip Aperture - LA), the tongue tip (Tongue Tip Constriction Degree - TTCD), and the velum (VEL). The average image intensity within these boxes serves as a proxy for constriction; as tissue moves into an ROI, the intensity increases, signaling a "closed" state.

The feature extraction process: (a) sample MRI image with ROIs, (b) masked image showing isolated regions, and the resulting multi-stream intensity data.

Statistical Modeling

The study compared four modeling approaches to classify these intensity streams into "Open" vs. "Closed" states:

  1. Vector Quantization (VQ): Simple threshold-based classification.
  2. Gaussian Mixture Models (GMM): Instantaneous statistical checks.
  3. Hidden Markov Models (HMM): Temporal models that smooth transitions within a single stream.
  4. Coupled Hidden Markov Models (CHMM): The most advanced approach, modeling three parallel chains (LA, TTCD, VEL) that interact. This allows the model to account for mechanical dependencies, distinguishing active gestures from passive movements.

Results and Discussion

The models were tested on rt-MRI recordings of a native English speaker producing phrases like "Type pay nova slowly" and "Type pain over slowly."

Model Performance

HMMs and CHMMs significantly outperformed VQ and GMMs. The latter two methods were prone to "spurious transitions," frequently flickering between states due to noise. In contrast, the Markovian models provided stable, linguistically meaningful segmentations.

Articulatory Timing

The CHMM framework successfully captured subtle timing lags inherent in nasal production. For the onset nasal in "nova," the average lag between velum opening and tongue tip closure was approximately 96.8ms. For the coda nasal in "pain over," this lag increased significantly to 279ms. These results demonstrate the model's ability to handle the natural asynchrony of speech articulators.

Conclusion

By treating mechanical coupling as a structural feature rather than noise, the CHMM approach provides a powerful tool for decoding the complex choreography of the vocal tract. Future work will focus on expanding the dataset to multiple speakers and implementing dynamic ROIs to account for head movement during recording.

Summary updated

BibTeX
@inproceedings{bresch2010statistical,
  author = {Erik Bresch and Athanasios Katsamanis and Louis Goldstein and Shrikanth S. Narayanan},
  title = {{Statistical multi-stream modeling of real-time MRI articulatory speech data}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-460},
}
Figure