75-Speaker Annot-16: A benchmark dataset for speech articulatory rt-MRI annotation with articulator contours and phonetic alignment

Xuan Shi, Yubin Zhang, Yijing Lu, Marcus Ma, Tiantian Feng, Asterios Toutios, Haley Hsu, Louis Goldstein, Shrikanth NarayananInterspeech 2025 · DOI

Introduction

Real-time MRI (rt-MRI) has significantly advanced our ability to observe speech production in vivo. However, the field faces a significant bottleneck: the lack of high-quality, labeled data. Raw MRI videos require precise annotations of "what" (articulatory contours) and "when" (phonetic timing) to be useful for machine learning. The 75-Speaker Annot-16 dataset addresses this by providing a robust, expert-verified benchmark subset of the widely used 75-Speaker vocal tract MRI database.

Methodology

The researchers developed a multi-stage annotation pipeline for 16 representative speakers (8 native American English, 8 non-native):

  1. Phonetic Alignment: Audio was synced with text using the Montreal Forced Aligner, followed by manual expert correction to ensure precise phoneme timing.
  2. Semi-Automatic Segmentation: A region-based algorithm in the Fourier domain was employed to track vocal tract movements. A custom MATLAB interface allowed experts to provide "seed" templates for automated tracking.
  3. Manual Ground Truths: Experts manually traced the contours of nine key articulators—including the tongue, lips, velum, epiglottis, and arytenoid—across 160 frames to serve as a validation "gold standard."
Anatomical landmarks and regions (R1, R2, R3) used for vocal tract segmentation and articulator tracking.

Dataset Characteristics

The 75-Speaker Annot-16 dataset is derived from the public 75-Speaker vocal tract MRI database, which features approximately 17 minutes of speech per participant at 83 frames per second. The new benchmark includes:

  • Detailed phonetic labels.
  • Automated contour tracks for the entire subset.
  • 160 high-quality, manually drawn ground-truth frames.

Key Insights and Innovations

The "SPAM" Model

A major contribution of this work is the development of SPAM (SAM2 for Speech Production), a fine-tuned version of Meta’s Segment Anything Model 2. While the base SAM2 model struggled with noisy, grayscale MRI data, the fine-tuned SPAM model demonstrated a significant leap in segmentation accuracy.

The Voicing Challenge

The study revealed a "voicing blind spot": models predicting phonemes solely from vocal tract shape struggled to distinguish between voiced/unvoiced pairs (e.g., /p/ vs. /b/). This is attributed to the vocal folds often being obscured or muddled in standard mid-sagittal MRI slices.

Linguistic Sensitivity (L1)

Baseline tests showed acoustic models are more accurate for native speakers (PER 0.288) than non-native speakers (PER 0.336), emphasizing the importance of diverse articulatory data for inclusive speech technology.

Results

The fine-tuned SPAM model achieved an Intersection over Union (IoU) of ~92% for seen subjects and ~84% for unseen subjects, a vast improvement over the 10-17% accuracy of the base SAM2. Furthermore, a Structured State Space (S4) model using only physical contours could predict the correct phoneme within its top three guesses over 51% of the time for unseen subjects.

Example of segmented regions in an rt-MRI frame, illustrating the output of the articulatory tracking process.

Conclusion and Future Work

While the current manual annotations cover 160 frames, the authors aim to scale these high-quality labels to the full 75-speaker set. Future work will focus on improving laryngeal imaging to address the voicing issue and exploring cross-modal learning to bridge the gap between audio and visual articulatory data.

Summary updated

BibTeX
@inproceedings{shi2025speaker,
  author = {Xuan Shi and Yubin Zhang and Yijing Lu and Marcus Ma and Tiantian Feng and Asterios Toutios and Haley Hsu and Louis Goldstein and Shrikanth Narayanan},
  title = {{75-Speaker Annot-16: A benchmark dataset for speech articulatory rt-MRI annotation with articulator contours and phonetic alignment}},
  booktitle = {Interspeech},
  year = {2025},
  doi = {10.21437/interspeech.2025-2394},
}
Figure