TL;DR
This paper introduces a groundbreaking framework for generating multi-speaker speech solely from silent electromyography (EMG) signals and facial images, eliminating the need for any audible input. This innovation holds significant promise for individuals with speech impairments by enabling them to communicate using their facial movements and a target speaker's voice identity.
What Problem Does This Paper Address?
Traditional silent speech interfaces (SSIs) often struggle with multi-speaker scenarios, as they typically require audible speech from the target speaker to capture their unique vocal characteristics (timbre). This limitation excludes individuals who cannot produce audible speech from benefiting fully. While recent advancements have enabled voice conversion using facial images, these methods still rely on audible source speech for content. This paper tackles the challenge of generating speech that is not only linguistically accurate but also matches a target speaker's voice identity, all without any audible input, making it a significant step towards more inclusive communication technologies.
Methods
The proposed framework integrates two key components: an EMG-based content estimation network and a face-based voice conversion network.
- EMG-based Content Estimation: Silent EMG signals, which capture articulatory muscle movements, are used to extract the linguistic content of speech. A novel "pitch-disentangled content embedding" is introduced to improve the extraction of this content from EMG, as silent EMG inherently lacks pitch information. This module is trained to generate a content embedding from EMG signals.
- Face-based Voice Conversion: This component leverages facial images of the target speaker to imbue the generated speech with their vocal identity. It comprises a face encoder for speaker embedding and a global pitch estimation network. The framework uses a conditional variational autoencoder architecture.
- Pitch-Flattening Module: A key innovation is the pitch-flattening module, applied before the content encoder. This module aims to remove content-related pitch variations from the ground-truth speech data used for training. The rationale is that silent EMG doesn't contain pitch information, and by "flattening" the training data's pitch, the model is forced to learn content-related pitch information more robustly, rather than relying on it being implicitly present in the content embedding.
The entire framework is trained end-to-end, with the content encoder utilizing pretrained SSL representations.
Datasets
- EMG Dataset: The Gaddy's dataset, containing approximately 20 hours of single-speaker EMG signals (both voiced and silent), was used for training the EMG-based content estimation network.
- Speech and Facial Image Dataset: The LRS3 dataset, comprising over 430 hours of TED and TEDx videos, was used to train the face-based voice conversion network and the face-based global pitch estimation network.
Key Insights and Ideas
- True Silent Speech Generation: The paper demonstrates the feasibility of generating multi-speaker speech entirely without audible inputs, a significant advancement for SSI technology.
- Facial Identity for Voice: The framework effectively uses facial images to capture and replicate a target speaker's vocal identity, even when no audio from that speaker is available.
- Pitch-Disentangled Content Embedding: The proposed pitch-disentangled content embedding is crucial for extracting meaningful linguistic information from EMG signals, which are inherently pitch-less.
- Pitch-Flattening for Robustness: The pitch-flattening module proves to be a surprisingly effective technique for improving the model's ability to generate natural, content-related pitch information, especially when dealing with silent EMG inputs. It forces the model to learn pitch more explicitly.
Key Results
- Intelligibility: The "Flatten" model (with pitch-flattening) showed improved intelligibility on silent EMG test sets compared to the "Base" model (without pitch-flattening), particularly for silent EMG signals.
- Consistency: Both objective and subjective consistency evaluations indicated that the model successfully captures target speaker characteristics. The Flatten model showed slightly better objective speaker embedding similarity.
- Pitch Deviations: The Flatten model significantly outperformed the Base model in reducing Local FO deviations, demonstrating its enhanced ability to generate natural, content-related pitch information. It also showed improvements in Global FO estimation.
Limitations and Future Work
- EMG Dataset Limitations: The current EMG dataset is from a single speaker, which might limit the diversity of content-related local pitch information. The authors anticipate greater gains from the pitch-flattening module on multi-speaker EMG datasets.
- Vocoder Quality: The vocoder component was trained on the LRS3 dataset, which contains relatively lower-quality data. Using pretrained vocoders or diffusion-based post-processing could potentially enhance generative quality.
- Pitch-Flattening Advancement: The pitch-flattening module used is a conventional signal processing method. Future work could explore more advanced pitch-controlling techniques for potentially more stable performance.
- Scope: The paper focuses on presenting the core framework and the pitch-flattening module. Further enhancements are left for future work.
Summary updated