3d tongue motion visualization based on ultrasound image sequences

Kele Xu, Yin Yang, A. Jaumard-Hakoun, Martine Adda-Decker, Angélique Amelot, Samer Al Kork, Lise Crevier-Buchman, Patrick Chawah, Gérard Dreyfus, Thibaut Fux, Claire Pillot-Loiseau, Pierre Roussel, Maureen Stone, Bruce DenbyInterspeech 2014 · OpenReview

TL;DR

This paper presents a novel real-time system for visualizing 3D tongue motion using ultrasound image sequences. By tracking speckle patterns in ultrasound images with a SIFT-based approach and then driving a 3D tongue model with this motion data, the system offers a more accessible and less invasive way to study speech production.

What Problem Does This Paper Address?

Understanding and visualizing human tongue motion is crucial for fields like speech production research, articulation training, and silent speech interfaces. However, directly measuring tongue movement is challenging due to its location within the oral cavity. Existing methods like MRI have limitations in temporal resolution and require subjects to be in a supine position, which is atypical for speech. X-ray imaging offers better temporal resolution but involves radiation exposure. Ultrasound imaging is a promising non-invasive, real-time alternative, but effectively extracting detailed tongue motion from these images and translating it into a dynamic 3D model has been a significant hurdle. Previous attempts using methods like Electromagnetic Articulography (EMA) have been invasive, potentially altering natural speech production.

Methods

The core of this work lies in two main components:

  1. Speckle Tracking for Motion Extraction: The authors leverage the inherent "speckle" patterns in ultrasound images, which are caused by the interference of scattered acoustic waves. While often considered noise, these patterns can act as unique signatures for tracking tissue movement. They employ a modified Scale-Invariant Feature Transform (SIFT) algorithm for this purpose. Unlike the standard SIFT, Gaussian filtering is omitted to preserve the fine speckle details crucial for tracking in ultrasound. This method detects and describes local invariant features (keypoints) within the speckle patterns. By matching these features between consecutive ultrasound frames, they can accurately track the displacement of points on the tongue's surface.
  2. 3D Tongue Model Driving: The motion data extracted from the ultrasound images (specifically, the displacements of mid-sagittal contour points) is used to drive a pre-existing 3D finite mesh model of the tongue (the ArtiSynth model). To achieve real-time visualization of tongue deformation, they utilize a technique called "modal warping." This method combines modal reduction with stiffness warping to efficiently simulate large, potentially rotational, deformations of the tongue model. Anchor nodes (fixed points representing anatomical landmarks like the hyoid bone) and constraint nodes (points on the mid-sagittal contour driven by ultrasound data) are used to guide the deformation of the 3D model.

Datasets

Not discussed in the paper. The paper focuses on a novel technique applied to ultrasound image sequences, implying that the data used is likely collected by the researchers themselves for demonstration purposes.

Key Insights and Ideas

  • Speckle as a Feature: The paper highlights that the seemingly noisy speckle patterns in ultrasound images are actually valuable for tracking subtle tissue movements, acting as natural markers.
  • SIFT Adaptation for Ultrasound: Modifying the SIFT algorithm by removing Gaussian filtering is a key insight for making it effective in the specific context of ultrasound speckle tracking. This preserves the fine details needed for accurate correspondence.
  • Mid-Sagittal Plane Focus: By concentrating on the mid-sagittal contour points from ultrasound images, the researchers simplify the problem while still capturing essential tongue movements relevant to speech articulation.
  • Modal Warping for Real-Time Deformation: The integration of modal warping is crucial for achieving real-time visualization of the complex, deformable tongue model, making the system practical for interactive applications.
  • Bridging Imaging and Modeling: The work effectively bridges the gap between real-time ultrasound imaging and biomechanical modeling, offering a more direct and less invasive way to visualize speech production compared to prior methods.

Key Results

The paper demonstrates a real-time system capable of visualizing 3D tongue motion driven by ultrasound sequences. While specific quantitative metrics like tracking accuracy or deformation error are not detailed, the authors present qualitative results through figures showing the speckle tracking correspondences and the driven 3D tongue model. They emphasize the system's potential for real-time application and its usefulness in various domains. The demonstration is expected to showcase the 3D tongue model motion visualization and an edit tool for modifying the tongue rest shape.

Limitations and Future Work

The authors acknowledge two primary limitations:

  • Speckle Tracking Stability: The performance of speckle tracking is not yet completely stable, particularly in segments of the ultrasound sequence where image quality is poor. This can lead to tracking inaccuracies.
  • Registration: The registration between the 2D ultrasound points and the 3D model nodes is still under development, indicating an area for further refinement.

Future work will likely focus on improving the robustness of the speckle tracking, refining the registration process, and further developing the user interface for applications like articulation training and subject-specific tongue modeling.

Summary updated

Abstract

The article proposes a real-time technique for visualizing tongue motion driven by ultrasound image sequences. Local feature description is used to follow characteristic speckle patterns in a set of mid-sagittal contour points in an ultrasound image sequence, which are then used as markers for describing movements of the tongue. A 3D tongue model is subsequently driven by the motion data extracted from the ultrasound image sequences. The “modal warping” technique is used for real-time tongue deformation visualization. The resulting system will be useful in a variety of domains including speech production study, articulation training, educational scenarios, etc. Some parts of the interface are still being developed; we will show preliminary results in the demonstration.
BibTeX
@inproceedings{xu2014d,
  author = {Kele Xu and Yin Yang and A. Jaumard-Hakoun and Martine Adda-Decker and Angélique Amelot and Samer Al Kork and Lise Crevier-Buchman and Patrick Chawah and Gérard Dreyfus and Thibaut Fux and Claire Pillot-Loiseau and Pierre Roussel and Maureen Stone and Bruce Denby},
  title = {{3d tongue motion visualization based on ultrasound image sequences}},
  booktitle = {Interspeech},
  year = {2014},
  url = {https://openreview.net/forum?id=bTdhNZ6fEM},
}
Figure