Impact of lack of acoustic feedback in EMG-based silent speech recognition

Matthias Janke, Michael Wand, Tanja SchultzInterspeech 2010

TL;DR

This paper explores how the absence of acoustic feedback affects EMG-based silent speech recognition. By analyzing spectral differences in EMG signals across audible, whispered, and silent speech, they developed a phoneme-aware spectral mapping technique that significantly improves silent speech recognition accuracy, demonstrating the importance of somatosensory feedback in silent articulation.

What Problem Does This Paper Address?

Traditional speech recognition systems rely on audible speech, which has limitations in noisy environments or for private communication. Silent Speech Interfaces (SSIs) offer a solution by using electromyography (EMG) to capture the electrical signals from articulatory muscles, bypassing the need for sound. While EMG-based recognition shows promise, a key challenge is understanding how the mode of speech production (audible vs. silent) impacts the EMG signal itself. Specifically, human speech production heavily relies on acoustic feedback. When this feedback is removed in silent speech, articulatory muscles might behave differently. This paper investigates these differences and proposes a method to compensate for them to improve silent speech recognition performance.

Methods

The core of this work lies in analyzing the spectral characteristics of EMG signals across different speaking modes and then using these insights to improve a speech recognition system.

  • EMG Signal Analysis: The researchers used Power Spectral Density (PSD) to quantify the "articulatory power" of EMG signals. They broke down signals into phoneme units and analyzed PSDs for consonants and vowels, and further subdivided these into bilabial/velar consonants and rounded/unrounded vowels. This allowed them to compare the spectral differences between audible, whispered, and silent speech.
  • Spectral Mapping Algorithm: Based on the analysis, they extended a previously developed spectral mapping algorithm. This new version incorporates phoneme-specific discrimination. For silent speech, they define a "base speaking mode" (audible or whispered) and a "mapping phonetic feature" (e.g., vowels). They compute a "mapping factor" based on the ratio of PSDs between the base mode and silent EMG for specific phoneme classes. This factor is then used to transform the frequency components of the silent EMG signal, effectively compensating for the perceived differences.
  • Speech Recognizer: The EMG-based speech recognition system uses three-state Hidden Markov Models (HMMs) to model phonetic features. They also employ a trigram language model for decoding.

Datasets

The study utilizes the EMG-UKA corpus, which was recorded from seven male and one female speaker. This corpus contains EMG signals for:

  • Audible speech
  • Whispered speech
  • Silent speech (mouthed)

Each speaker recorded parallel utterances across these three modes. The corpus consists of 243 minutes of recordings. For recognition experiments, 40 SPEC utterances were used for training, and 10 BASE utterances were used for testing. This corpus is noted as being comparable to the EMG-PIT corpus but includes whispered speech, which was deemed essential for this study.

Key Insights and Ideas

  • Somatosensory Feedback Compensation: The lack of acoustic feedback in silent speech appears to be compensated by increased reliance on somatosensory feedback (tactile and proprioceptive sensations). This means speakers might articulate sounds that provide stronger tactile feedback more forcefully when speaking silently.
  • Phoneme-Specific Articulation Differences: The spectral differences between speaking modes are not uniform across all phonemes.
  • Vowels: Vowel articulation in audible speech generally shows higher spectral density. In silent speech, rounded vowels (which involve more lip movement) show relatively stronger articulation than unrounded vowels, supporting the somatosensory feedback hypothesis.
  • Consonants: Bilabial consonants (like 'p', 'b', 'm'), which involve direct lip closure, are articulated more strongly in silent speech compared to audible or whispered speech. This contrasts with velar consonants ('g', 'k', 'ng'), where the articulation power decreases from audible to silent speech. This suggests that sounds with more direct tactile feedback at the lips are hyperarticulated in silent speech.
  • Whispered Speech as an Intermediate: Whispered speech often falls spectrally between audible and silent speech, suggesting it might offer some limited tactile feedback while still being relatively quiet.

Key Results

The primary quantitative result is the improvement in silent speech recognition accuracy using the proposed spectral mapping method:

  • Baseline Word Error Rate (WER): 61.67%
  • Consonant-Mapping WER: 57.16% (a relative improvement of 7.31%)
  • Vowel-Mapping WER: 54.52% (a relative improvement of 11.59%)

The vowel-mapping approach yielded the most significant improvement, highlighting the importance of compensating for vowel articulation differences in silent speech.

Limitations and Future Work

  • Limitations:
    • The quality of time-alignments for silent EMG can be affected by the discrepancy between audible and silent speech articulation, although the authors state their chosen method is the most suitable found so far.
  • The analysis of PSD ratios was performed on specific channels (2 and 4) closest to lip-controlling muscles, assuming lip movement is a primary source of somatosensory feedback. Other muscle groups might contribute differently.
  • Future Work: The paper suggests that this phoneme-based spectral mapping can enable seamless switching between audible and silent speaking modes within a single recognition system, which is a promising direction for more robust and versatile SSIs. Further exploration into other phonetic feature distinctions and their impact on somatosensory feedback could lead to even more refined compensation strategies.

Summary updated

BibTeX
@inproceedings{janke2010impact,
  author = {Matthias Janke and Michael Wand and Tanja Schultz},
  title = {{Impact of lack of acoustic feedback in EMG-based silent speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
}

Loading related content...

Figure