Across-speaker articulatory normalization for speaker-independent silent speech recognition

Jun Wang, Ashok Samal, Jordan R. GreenInterspeech 2014

TL;DR

This paper introduces a novel method for "silent speech recognition" (SSR) that significantly improves accuracy by normalizing articulatory data across different speakers. By using Procrustes matching to align tongue and lip movements, the system achieves a remarkable jump in speaker-independent recognition accuracy, paving the way for more accessible and robust SSR technologies.

What Problem Does This Paper Address?

Silent Speech Interfaces (SSIs) aim to enable communication for individuals who cannot speak due to conditions like laryngectomy or neurological disorders. These systems work by recognizing speech from articulatory movements (tongue, lips) rather than audio. A major hurdle in developing effective SSIs is the "across-speaker variation" – the fact that people move their articulators differently due to individual anatomy, habits, and co-articulation patterns. Current SSI approaches often rely on speaker-dependent models, requiring extensive training data for each user, which is impractical for individuals with speech impairments. This paper tackles the challenge of creating speaker-independent SSI models that can generalize across users without needing extensive per-user training, thereby reducing the data burden and increasing accessibility.

Methods

The core of this paper's contribution lies in its across-speaker articulatory normalization technique, based on Procrustes matching. Procrustes analysis is a statistical method for comparing shapes by removing differences in location, scale, and rotation. In this context, articulatory movements (represented as sequences of 3D coordinates from sensors on the tongue and lips) are treated as "shapes."

The normalization process involves three key steps:

  1. Translation: Shifting each speaker's articulatory data so that the centroid (average position) of the movement shape is at the origin (0,0). This removes locational differences.
  2. Scaling: Adjusting the size of each shape to a unit size. This accounts for variations in vocal tract and tongue sizes between speakers.
  3. Rotation: Aligning the shapes so that a reference axis (defined by the average positions of the upper and lower lip movements) is vertical. This corrects for differences in facial anatomy and how speakers orient their movements.

After normalization, the processed articulatory data is fed into a Support Vector Machine (SVM) classifier, specifically using a Radial Basis Function (RBF) kernel. SVMs are well-suited for finding decision boundaries between different classes of data, in this case, different spoken phrases.

Datasets

The study utilized a custom dataset collected from seven native English speakers. These participants produced a set of twelve short functional phrases. The articulatory data was captured using an Electromagnetic Articulograph (EMA) AG500, which records the 3D motion of sensors attached to the tongue and lips at a sampling rate of 200 Hz. The paper notes that while the EMA system is currently cumbersome, the normalization approach itself is device-agnostic and can be applied to future portable devices.

Key Insights and Ideas

  • Articulatory Variation is a Major Bottleneck: The significant drop in accuracy when moving from speaker-dependent to speaker-independent models (68.63% vs. 94.31%) clearly highlights how much inter-talker variability hinders recognition.
  • Procrustes Matching is a Powerful Tool for Shape Normalization: The technique effectively "standardizes" articulatory shapes, making them more comparable across individuals. This is crucial for building generalized models.
  • Normalization Bridges the Gap: The dramatic improvement in speaker-independent accuracy after normalization (from 68.63% to 95.90%) demonstrates that Procrustes matching can effectively mitigate the negative effects of inter-talker variation.
  • Potential for Truly Accessible SSIs: By enabling speaker-independent recognition, this approach reduces the need for extensive per-user training data, making SSIs more practical for individuals with speech impairments who may have limited ability to provide such data.

Key Results

The study reported a substantial increase in classification accuracy:

  • Speaker-dependent accuracy: 94.31%
  • Speaker-independent (unnormalized) accuracy: 68.63%
  • Speaker-independent (normalized) accuracy: 95.90%

The improvement from the unnormalized speaker-independent approach to the normalized one was statistically significant (p < 0.01), indicating the effectiveness of the Procrustes matching normalization. The normalized speaker-independent accuracy even surpassed the speaker-dependent accuracy, suggesting that normalization can potentially lead to even better performance than traditional speaker-dependent models in some scenarios.

Limitations and Future Work

The authors acknowledge several limitations:

  • Small Dataset Size: The experiment used a limited set of phrases and a relatively small number of participants.
  • Cumbersome EMA System: The current EMA technology is too bulky for widespread clinical use, though the normalization method itself is device-agnostic.

Future work suggested includes:

  • Expanding Vocabulary: Testing the approach with a larger set of phonemes and words.
  • Larger Subject Pool: Including subjects with diverse genders and dialects.
  • Real-time Implementation: Integrating the normalization into an online SSI.
  • Clinical Application: Evaluating the efficacy of the approach for individuals with speech impairments (e.g., post-laryngectomy patients).
  • Individual Articulator Normalization: Exploring normalization techniques for specific articulators.

Summary updated

BibTeX
@inproceedings{wang2014across,
  author = {Jun Wang and Ashok Samal and Jordan R. Green},
  title = {{Across-speaker articulatory normalization for speaker-independent silent speech recognition}},
  booktitle = {Interspeech},
  year = {2014},
}
Figure