Using Ultrasound Imaging to Create Augmented Visual Biofeedback for Articulatory Practice

Colin T. Annand, Maurice Lamb, Sarah Dugan, Sarah R. Li, Hannah M. Woeste, T. Douglas Mast, Michael A. Riley, Jack A. Masterson, Neeraja Mahalingam, Kathryn J. Eary, Caroline Spencer, Suzanne Boyce, Stephanie Jackson, Anoosha Baxi, Reneé SewardInterspeech 2019

TL;DR

This paper introduces a novel method for creating augmented visual biofeedback for speech therapy using real-time ultrasound tongue tracking. By translating complex tongue movements into intuitive game controls, they aim to improve pronunciation practice, particularly for challenging sounds like the American English /r/.

What Problem Does This Paper Address?

Practicing speech sounds, especially those that are difficult to produce (like the American English /r/ for some children or second-language speakers), often relies on clinical feedback. While ultrasound imaging provides a detailed view of tongue movement, interpreting this complex, rapid motion can be challenging for both clinicians and speakers. The authors aim to bridge this gap by simplifying the visual feedback, making it more intuitive and engaging for speakers to learn and adjust their articulatory patterns. They propose an augmented biofeedback system that translates tongue movements into a game-like experience, making the practice more motivating and the feedback more directly actionable.

Methods

The core of their method is a real-time tracking program called TonguePART. This program takes ultrasound images of the tongue and, after an initial manual calibration, automatically tracks the tongue contour at 36 frames per second. The tongue contour is then divided into three equal regions: root, dorsum, and blade. For each region, a reference point is identified, and its average position is calculated. These regional displacements are transmitted via TCP/IP to a feedback program.

The augmented visual biofeedback is demonstrated through a "basketball shooting" paradigm. A prototype game, developed in Unity3D, uses the real-time tongue tracking data to control game elements. Specifically, the movement of the tongue regions can be mapped to control the basketball's trajectory (along X, Y, or Z axes). The goal is to make the basketball shot by conforming to target tongue trajectories derived from accurate /r/ productions. This approach aims to provide an intuitive, externally focused control signal for articulatory practice.

Datasets

Not discussed in the paper. The paper mentions using sample /r/ productions from speakers with and without Residual Speech Sound Disorders (RSSD) for demonstration purposes, but does not detail specific datasets used for training or evaluation.

Key Insights and Ideas

  • Simplifying complex feedback: The core idea is that translating complex, internal articulatory movements into a simple, external game goal (like making a basket) can be more effective for learning than direct instruction or opaque visual feedback.
  • Gamification for motivation: Using a game-like paradigm can significantly increase motivation for practicing speech sounds, especially for individuals who might find traditional methods tedious or difficult to understand.
  • Implicit learning through external focus: The approach leverages motor learning principles that suggest an external focus of attention (e.g., controlling a game character) can lead to more effective and implicit learning of new motor patterns.
  • Potential for individualized therapy: The system's parameters can be adjusted to accommodate individual variations in tongue movement, allowing for personalized therapy that still requires specific articulatory configurations for success.

Key Results

The paper presents a conceptual demonstration and a prototype game. While quantitative results are not detailed, the authors describe the successful real-time tracking of tongue contours and the transmission of this data to control game parameters. The "Show and Tell" demonstration will feature real-time feedback based on sample /r/ productions, illustrating how different trajectories can be generated. The clinical applicability is highlighted by the potential to guide speakers towards desired tongue movements by adjusting the game's difficulty (e.g., requiring greater articulatory displacement for a successful shot).

Limitations and Future Work

The authors acknowledge that this is a conceptual demonstration and a prototype. Future work includes implementing this concept in a real-time biofeedback system within a clinical setting. They also suggest that the target /r/ productions may involve quasi-independent motion of tongue regions, which their method is designed to capture and potentially leverage for clinically relevant control signals. The paper implies that further research is needed to fully validate the effectiveness of this augmented biofeedback approach in clinical practice.

Summary updated

BibTeX
@inproceedings{annand2019using,
  author = {Colin T. Annand and Maurice Lamb and Sarah Dugan and Sarah R. Li and Hannah M. Woeste and T. Douglas Mast and Michael A. Riley and Jack A. Masterson and Neeraja Mahalingam and Kathryn J. Eary and Caroline Spencer and Suzanne Boyce and Stephanie Jackson and Anoosha Baxi and Reneé Seward},
  title = {{Using Ultrasound Imaging to Create Augmented Visual Biofeedback for Articulatory Practice}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure