A hybrid approach to 3d tongue modeling from vocal tract MRI using unsupervised image segmentation and mesh deformation

Alexander Hewer, Ingmar Steiner, Stefanie WuhrerInterspeech 2014

TL;DR

This paper introduces a novel hybrid approach to automatically generate 3D tongue models from vocal tract MRI scans. By combining unsupervised image segmentation with a mesh deformation technique, the method can create realistic tongue shapes even from limited 2D data, paving the way for more accurate speech production analysis and synthesis.

What Problem Does This Paper Address?

Understanding how the tongue moves during speech is crucial for fields like speech science, phonetics, and computer graphics (e.g., for realistic animated characters or pronunciation training tools). Magnetic Resonance Imaging (MRI) has become a leading non-invasive technique for capturing vocal tract dynamics. However, raw MRI data is complex and requires segmentation to isolate specific structures like the tongue. Existing methods often rely on extensive manual annotation, are limited to 2D, or require specialized training data, making them less flexible and scalable. This paper aims to bridge this gap by offering a more automated and adaptable approach to extract 3D tongue models from MRI, even from less ideal data sources.

Methods

The core of the proposed method is a two-step process:

  1. Unsupervised Image Segmentation: The first step involves segmenting the tongue from the MRI scan. The authors propose using either the Chan-Vese algorithm or a graph cut algorithm.
  2. The Chan-Vese algorithm is a global method that evolves a contour to minimize the variance of gray values within segmented regions. While it can be minimally supervised (requiring an initial contour), the authors use it in an unsupervised manner by always initializing it with an axis-aligned ellipsoid.
  3. The graph cut algorithm is presented as a robust fallback. It's a local method that uses neighborhood information to find the optimal partition. It requires minimal user input in the form of two annotation sets (points to include and exclude from the tongue region).
  4. Both segmentation methods produce a point cloud representing the tongue's surface.
  5. Mesh Deformation: The extracted point cloud is then used to deform a generic 3D template mesh. This deformation process is designed to be robust even with sparse point clouds.
  6. The technique used is based on minimizing an energy function that balances fitting the point cloud data with maintaining the mesh's shape to avoid distortions.
  7. Crucially, this step relies exclusively on geometric information, meaning the same generic template mesh can be used across different scans and speakers, regardless of variations in scanner or individual anatomy.

Datasets

The authors evaluated their approach on three distinct datasets:

  • Static 3D MRI (Baker [14]): This dataset contains static 3D MRI scans of a male speaker producing various phonemes and speech configurations. It includes sagittal slices and identified mid-sagittal slices.
  • Movie of Real-time 2D MRI (Niebergall et al. [15]): This is a video of real-time 2D MRI showing the mid-sagittal slice of a speaker. The authors processed individual frames from this video.
  • Corpus of real-time 2D MRI (USC-TIMIT database [16]): This dataset consists of updated dynamic 2D MRI recordings with a higher temporal resolution. The authors also processed individual frames from this dataset, noting issues with vignetting artifacts.

No new benchmarks were introduced; existing publicly available datasets were utilized.

Key Insights and Ideas

  • Geometric-only Mesh Deformation: The most significant insight is the decoupling of the mesh deformation from specific appearance cues (like color or intensity). By relying solely on geometric information, the generic template mesh can adapt to diverse speakers and scanning conditions without needing speaker-specific training.
  • Hybrid Approach Robustness: Combining image segmentation (even minimally supervised) with mesh deformation provides a powerful hybrid approach. This allows for flexibility in choosing the best segmentation method for a given scan and ensures that even sparse or noisy point clouds can yield a usable 3D model.
  • 2D Data Potential: The paper demonstrates that realistic 3D tongue shapes can be extracted even from 2D real-time MRI frames. This opens up possibilities for analyzing speech dynamics using more accessible and faster MRI sequences.
  • Segmentation Challenges: The authors highlight that certain phonemes (like /k/ or /q/) pose challenges for segmentation due to tongue-palate contact obscuring boundaries and the tongue shape deviating significantly from the generic template.

Key Results

  • Quantitative: For static 3D MRI data, approximately 70% of the vertex errors (distance to the nearest point in the point cloud) were below 2 mm. Results for 2D data were slightly worse but still considered acceptable.
  • Qualitative: The method produced "promising results" and "satisfying" tongue shapes across the tested datasets. The paper includes figures showing example segmentation and deformed meshes.
  • Limitations: The method struggled with phonemes where tongue-palate contact obscured boundaries. Temporal noise was observed in the 2D real-time MRI sequences due to the lack of temporal coherence exploitation. Vignetting artifacts in some 2D scans also impacted segmentation.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Problematic Phonemes: To address issues with phonemes like /k/, they plan to reconstruct missing palate information by incorporating data from different vocal tract configurations or external articulography (EMA) measurements.
  • Statistical Models: They aim to train a statistical 3D tongue model to improve template matching, especially for sparse 2D data, drawing inspiration from prior work on predicting 3D shapes from 2D contours.
  • Combining Modalities: For 2D data, they want to combine lateral EMA information with mid-sagittal MRI scans to better inform the alignment of the 3D mesh.
  • Temporal Smoothing: To mitigate temporal noise in real-time 2D MRI, they plan to apply temporal smoothing techniques.
  • Inter-frame Information Sharing: They propose using the result of the previous frame as the template for the current frame in real-time sequences to leverage temporal coherence and improve alignment.

Summary updated

BibTeX
@inproceedings{hewer2014hybrid,
  author = {Alexander Hewer and Ingmar Steiner and Stefanie Wuhrer},
  title = {{A hybrid approach to 3d tongue modeling from vocal tract MRI using unsupervised image segmentation and mesh deformation}},
  booktitle = {Interspeech},
  year = {2014},
}
Figure