Introduction
Real-time MRI (rt-MRI) has significantly advanced our ability to observe speech production in vivo. However, the field faces a significant bottleneck: the lack of high-quality, labeled data. Raw MRI videos require precise annotations of "what" (articulatory contours) and "when" (phonetic timing) to be useful for machine learning. The 75-Speaker Annot-16 dataset addresses this by providing a robust, expert-verified benchmark subset of the widely used 75-Speaker vocal tract MRI database.
Methodology
The researchers developed a multi-stage annotation pipeline for 16 representative speakers (8 native American English, 8 non-native):
- Phonetic Alignment: Audio was synced with text using the Montreal Forced Aligner, followed by manual expert correction to ensure precise phoneme timing.
- Semi-Automatic Segmentation: A region-based algorithm in the Fourier domain was employed to track vocal tract movements. A custom MATLAB interface allowed experts to provide "seed" templates for automated tracking.
- Manual Ground Truths: Experts manually traced the contours of nine key articulators—including the tongue, lips, velum, epiglottis, and arytenoid—across 160 frames to serve as a validation "gold standard."
Dataset Characteristics
The 75-Speaker Annot-16 dataset is derived from the public 75-Speaker vocal tract MRI database, which features approximately 17 minutes of speech per participant at 83 frames per second. The new benchmark includes:
- Detailed phonetic labels.
- Automated contour tracks for the entire subset.
- 160 high-quality, manually drawn ground-truth frames.
Key Insights and Innovations
The "SPAM" Model
A major contribution of this work is the development of SPAM (SAM2 for Speech Production), a fine-tuned version of Meta’s Segment Anything Model 2. While the base SAM2 model struggled with noisy, grayscale MRI data, the fine-tuned SPAM model demonstrated a significant leap in segmentation accuracy.
The Voicing Challenge
The study revealed a "voicing blind spot": models predicting phonemes solely from vocal tract shape struggled to distinguish between voiced/unvoiced pairs (e.g., /p/ vs. /b/). This is attributed to the vocal folds often being obscured or muddled in standard mid-sagittal MRI slices.
Linguistic Sensitivity (L1)
Baseline tests showed acoustic models are more accurate for native speakers (PER 0.288) than non-native speakers (PER 0.336), emphasizing the importance of diverse articulatory data for inclusive speech technology.
Results
The fine-tuned SPAM model achieved an Intersection over Union (IoU) of ~92% for seen subjects and ~84% for unseen subjects, a vast improvement over the 10-17% accuracy of the base SAM2. Furthermore, a Structured State Space (S4) model using only physical contours could predict the correct phoneme within its top three guesses over 51% of the time for unseen subjects.
Conclusion and Future Work
While the current manual annotations cover 160 frames, the authors aim to scale these high-quality labels to the full 75-speaker set. Future work will focus on improving laryngeal imaging to address the voicing issue and exploring cross-modal learning to bridge the gap between audio and visual articulatory data.
Summary updated