Speech Organ Contour Extraction Using Real-Time MRI and Machine Learning Method

Hironori Takemoto, Tsubasa Goto, Yuya Hagihara, Sayaka Hamanaka, Tatsuya Kitamura, Yukiko Nota, Kikuo MaekawaInterspeech 2019

TL;DR

This paper introduces a machine learning approach to automatically extract speech organ contours from real-time MRI videos. By training a model on manually traced contours, they achieved higher accuracy than human operators, even when applied to new videos from the same subject, suggesting a promising avenue for efficient and accurate articulatory movement analysis.

What Problem Does This Paper Address?

Analyzing the intricate movements of speech organs during articulation is crucial for understanding speech production. Real-time Magnetic Resonance Imaging (rtMRI) offers a powerful way to capture these movements as videos. However, extracting precise contours of organs like the tongue, lips, and palate from these videos is challenging. The boundaries between air and tissue can be unclear due to noise, and manual tracing is time-consuming and prone to inter-operator variability. This paper aims to automate this contour extraction process using machine learning, thereby improving efficiency and consistency.

Methods

The core of this paper is the application of a machine learning method, specifically using the Dlib library, to extract speech organ contours. The process involves:

  1. Training Data Generation: Expert operators manually traced the contours of five specific speech organs (tongue, lips, palates, p-wall, a-wall) from selected frames of an rtMRI video. These manually traced contours served as the "ground truth" for training.
  2. Machine Learning Model Training: For each speech organ, a separate "learner" (a machine learning model) was trained using the manually traced contours. The paper mentions using a random forest algorithm, a common and effective technique for this type of task.
  3. Contour Extraction: Once trained, these learners were used to automatically extract contours from new frames of the rtMRI videos.
  4. Accuracy Evaluation: The accuracy of the machine learning-extracted contours was evaluated by comparing them to the ground truth. This comparison was done by calculating the geometrical distance between the machine learning contours and the ground truth, as well as comparing them to contours traced by other human operators (both expert and non-expert).

The novelty lies in adapting existing machine learning tools (originally for facial landmark detection) to the specific domain of speech organ contour extraction from rtMRI data and demonstrating its effectiveness.

Datasets

  • Training Data: Manually traced contours of five speech organs (tongue, lips, palates, p-wall, a-wall) from selected frames of one rtMRI video (video 15) of a single male Japanese speaker.
  • Evaluation Data:
    • Contours from frames not used in the training set of video 15, traced by two additional expert operators and three non-expert operators, served as ground truth for initial accuracy assessment.
  • For evaluating the generalizability of the trained learners, rtMRI videos 5, 10, 15, 20, 25, 35, 45, and 55 were used. Within these videos, specific frames (128, 256, and 384) were selected for analysis.

Key Insights and Ideas

  • Machine Learning Outperforms Humans: The machine learning approach consistently achieved lower error rates than both expert and non-expert human operators across all speech organs. This suggests that the trained models can capture articulatory nuances more reliably than manual tracing, especially for complex or difficult-to-trace regions.
  • Consistency Across Organs: A significant finding is the relatively constant error rate of the machine learning method across different speech organs. In contrast, human operators showed much greater variability in error, likely due to the inherent difficulty in tracing certain organs.
  • Generalizability to New Videos: Learners trained on a single video (video 15) demonstrated good accuracy when applied to other videos from the same subject, even if those videos were recorded later and potentially with slight variations in head orientation. This is a crucial insight for practical applications, as it reduces the need for extensive retraining.
  • Effectiveness in Difficult Regions: Machine learning proved particularly effective in areas where manual tracing is challenging, such as the a-wall and p-wall. This highlights its potential to overcome limitations of human perception and manual dexterity.

Key Results

  • Lower Error Rates: Machine learning achieved significantly lower error values (geometrical distance in pixels) compared to both ex-operators and nex-operators for all speech organs, with the exception of the tongue where the difference was less pronounced.
  • Consistent Performance: The error for machine learning was nearly constant across the five speech organs, whereas human operator errors varied significantly, increasing for the tongue, lips, palates, p-wall, and a-wall in that order.
  • Minimal Impact of New Videos: When learners trained on video 15 were applied to other videos, the error variations were generally not significant. Only the lips in specific videos showed a statistically significant increase in error, which the authors attribute to the lip contour having the lowest error in the training video itself.
  • Efficiency: Training a learner took approximately 90 seconds. Extracting contours for 512 frames from a video took about 15 seconds. This represents a substantial time saving compared to manual tracing, which required about 20 minutes per frame for an expert.

Limitations and Future Work

  • Single Subject: The study was conducted on a single male Japanese speaker. The authors acknowledge that the generalizability of the trained learners to other subjects was not examined and suggest this as a direction for future work.
  • Head Orientation Changes: While the learners showed robustness to slight head orientation changes, drastic changes between videos might necessitate building training data sets using multiple videos.
  • Overlapping Contours: When speech organs are in close contact (e.g., soft palate and pharyngeal wall, or tongue and palate), the machine learning approach sometimes struggled with independent contour extraction, leading to overlaps. The authors propose post-processing solutions to address this.
  • Future Work: The authors suggest exploring the application of these methods to videos from different subjects and further developing the approach for more accurate tracking of articulatory movements.

Summary updated

BibTeX
@inproceedings{takemoto2019speech,
  author = {Hironori Takemoto and Tsubasa Goto and Yuya Hagihara and Sayaka Hamanaka and Tatsuya Kitamura and Yukiko Nota and Kikuo Maekawa},
  title = {{Speech Organ Contour Extraction Using Real-Time MRI and Machine Learning Method}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure