TL;DR
This paper explores synthesizing speech from facial muscle activity (EMG) using voice transformation (VT) techniques, aiming for silent speech interfaces. They adapt standard VT methods to map EMG signals to speech features, showing preliminary success and paving the way for new communication technologies.
What Problem Does This Paper Address?
Communicating silently is a significant challenge, especially in noisy environments or when privacy is paramount. Traditional speech recognition relies on audible sound, which is unavailable in silent speech scenarios. While electromyography (EMG) can capture the electrical signals from facial muscles involved in speech articulation, effectively translating these signals into intelligible speech has been a hurdle. Previous attempts often focused on Automatic Speech Recognition (ASR) of EMG, but this paper investigates a different approach: synthesizing actual speech from EMG data, making it usable by both ASR systems and human listeners. This opens doors for silent speech interfaces that are more versatile and accessible.
Methods
The core of this research lies in adapting Voice Transformation (VT) techniques, typically used to change one person's voice to another's, for EMG-to-speech synthesis. Specifically, they leverage Gaussian Mixture Model (GMM) mapping techniques, a standard in VT, to learn the relationship between EMG-derived features and speech parameters.
Key methodological adaptations include:
- Data Alignment: Instead of relying on standard time-warping techniques for aligning speech signals, they use an offset based on marker signals, as direct frame-based comparison between EMG and audio features is difficult.
- Feature Mapping: EMG features are used as the "source" input for the VT system, while spectral features are extracted from the target (audible) speech.
- Handling Excitation: For silent speech, where no audio is available for pitch (Fo) and power estimation, they explore treating utterances as unvoiced (using noise as excitation) or synthesizing monotonous speech. Power is also treated as a predictable feature from EMG.
- System Adaptation: The FestVox GMM mapping-based VT system is modified to handle the unique characteristics of EMG data.
Datasets
The authors utilized three EMG datasets:
- 100.007: Audible EMG recordings from a single male speaker with a Taiwanese English accent. Split into training (380 sentences) and testing (120 sentences).
- 100.018: Audible and silent EMG recordings from the same speaker as 100.007. Training set included 190 audible and 190 silent sentences; test set had 60 audible and 60 silent utterances.
- EMG-PIT: Data from 14 female speakers, collected as part of a psychobiological study on stress and voice. Each session included 100 sentences (half audible, half silent). Individual sentences were used for training, and "base" sentences for testing.
For validation of their VT system, they also used the MOCHA database's EMA-to-speech data.
Key Insights and Ideas
- VT as a Viable Path for EMG-to-Speech: The research demonstrates that voice transformation, a technique not originally designed for EMG, can be successfully adapted to synthesize speech from muscle activity. This broadens the application of VT beyond voice identity conversion.
- EMG Signals Contain Rich Spectral Information: The preliminary results suggest that EMG signals, even when used to predict only spectral features, contain enough information to produce somewhat intelligible speech.
- Silent Speech Presents Unique Challenges: Synthesizing speech from silent EMG is significantly harder than from audible EMG. The lack of natural excitation (pitch and voicing) signals is a major obstacle, requiring novel approaches like using noise or synthesized pitch.
- Differences Between Audible and Silent EMG: The study observed differences in EMG signal characteristics between audible and silent speech, particularly in power and power trajectory. This suggests that separate models or adaptations might be needed for each.
- Potential for Improved ASR: The synthesized EMG-to-speech utterances performed surprisingly well when fed into an ASR system, even outperforming direct ASR from EMG features in some cases. This highlights the potential of synthesized speech as an intermediate representation.
Key Results
- EMA-to-Speech Validation: Their adapted VT system achieved an average Mel-Cepstral Distortion (MCD) of 4.46 (std. dev. 1.67) on EMA-to-speech, slightly outperforming previous work.
- EMG-to-Speech (Audible): For EMG-to-speech using audible data, MCD values were higher than EMA-to-speech but still yielded "fairly understandable" utterances, with sibilants noted as clearer than in EMA-to-speech.
- ASR on Synthesized Speech: When tested with an ASR system, the EMG-to-speech utterances achieved a Word Error Rate (WER) of 15.7% (with 4 Gaussian components), which was better than direct ASR from EMG features (18.0% WER).
- EMG-to-Whisper: Informal listening tests for EMG-to-whisper synthesis (for silent speech) found the output mostly unintelligible, though some words were understandable. ASR on EMG-to-whisper from audible speech yielded WERs from 33.3% to 91.90%, and from silent speech, 79.8% to 94.90%.
Limitations and Future Work
- Limited Phonetic Coverage: The EMG-PIT corpus, while from multiple speakers, had a smaller number of unique sentences per speaker, potentially limiting the phonetic coverage for training.
- Excitation Signal Generation: A significant challenge remains in generating adequate and natural excitation signals (pitch and voicing) for silent speech synthesis.
- EMG Signal Quality and Electrode Placement: The authors acknowledge that the sufficiency of the five EMG channels and their placement for fully representing speech is still an open question.
- Audible vs. Silent Speech Differences: Further investigation is needed to fully understand and model the differences between EMG signals produced during audible and silent speech.
- Future Directions:
- Improving excitation signal generation for silent speech.
- Exploring different EMG feature representations.
- Investigating optimal electrode placement.
- Developing real-time EMG synthesis systems to allow users to adjust their silent speech based on auditory feedback.
- Conducting formal listening evaluations for synthesized speech.
Summary updated