Whole-Word Recognition from Articulatory Movements for Silent Speech Interfaces

Jun Wang, Ashok Samal, Jordan R. Green, Frank Rudzicz

Introduction

For individuals who have undergone a laryngectomy or suffer from severe speech impairments, restoring natural communication is a critical challenge. Traditional methods such as esophageal speech or the use of an electrolarynx often fall short, providing limited intelligibility or an unnatural, robotic tone. Silent Speech Interfaces (SSIs) offer a transformative alternative by converting non-acoustic articulatory movements—such as those of the tongue and lips—directly into synthesized speech or text.

A major hurdle in SSI development has been the "granularity" of recognition. Phoneme-level systems frequently struggle with coarticulation, where the physical movement for a specific sound is altered by its neighbors. Conversely, sentence-level systems, while accurate, lack the flexibility to handle novel combinations of words. This research proposes a "sweet spot": a real-time, whole-word recognition system that leverages the robustness of word-level units to overcome these limitations.

System Architecture and Methodology

The proposed system utilizes an end-to-end pipeline designed for low-latency, high-accuracy performance. The process begins with data acquisition using an Electromagnetic Articulograph (EMA), which tracks the 3D coordinates of sensors placed on the speaker's tongue and lips.

System Architecture

Unlike traditional approaches that attempt to extract complex "Articulatory Features," this model employs a direct mapping strategy. Raw, time-normalized spatial data is fed into a Support Vector Machine (SVM) classifier with a Radial Basis Function (RBF) kernel. This approach simplifies the processing pipeline and minimizes error propagation.

For continuous speech, the system implements an online recognition algorithm using a variable-size moving window. It continuously calculates the probability of various word matches. To ensure reliability, the algorithm uses dynamic thresholding and temporal proximity constraints, preventing the system from predicting overlapping or impossible word sequences.

Word Probability Trends

Experimental Results

The algorithm was evaluated using a dataset of 5,500 isolated word samples from 10 healthy speakers. The results demonstrate the system's potential for real-time clinical application:

  • Accuracy: The SVM-based classifier achieved a high classification accuracy of 93.71% for isolated words.
  • Sequence Performance: In continuous 25-word sequences, the system maintained high fidelity, missing an average of only 1.93 words per sequence.
  • Latency: Real-time viability is a core strength; the average latency for word prediction was 0.79 seconds, with an average temporal offset of just 142ms from the actual articulation.
  • Efficiency: The SVM approach proved significantly more efficient than standard Dynamic Time Warping (DTW), processing words in 0.5ms compared to DTW's 1287.5ms.

Discussion and Future Directions

While the current results are promising, several steps remain before clinical deployment. The study was conducted with a limited vocabulary and healthy participants. Future work will focus on expanding the system to a core vocabulary of at least 250 words and validating the algorithm with laryngectomy patients, whose articulatory patterns may differ significantly. Additionally, transitioning from bulky EMA hardware to more portable solutions, such as ultrasound or wearable sensors, will be essential for user comfort and daily utility.

By focusing on whole-word units and leveraging efficient machine learning models, this research provides a robust framework for the next generation of Silent Speech Interfaces, bringing fluid, natural communication closer to reality for those without a voice.

Summary updated

BibTeX
@inproceedings{wang2012whole,
  author = {Jun Wang and Ashok Samal and Jordan R. Green and Frank Rudzicz},
  title = {{Whole-Word Recognition from Articulatory Movements for Silent Speech Interfaces}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-318},
}
Figure