An Improved Model for Voicing Silent Speech

David Gaddy, Dan KleinACL 2021

TL;DR

This paper introduces a novel deep learning model that synthesizes intelligible speech from silent facial muscle movements (EMG signals). By leveraging convolutional neural networks for feature extraction and a Transformer architecture for temporal modeling, coupled with an auxiliary phoneme prediction task, the model achieves a significant 25.8% absolute reduction in word error rate on an open-vocabulary task, outperforming previous state-of-the-art by a substantial margin.

What Problem Does This Paper Address?

The core challenge this paper tackles is the synthesis of intelligible speech from silent speech, specifically using electromyography (EMG) signals captured from facial muscles. While previous work has shown promise in narrow vocabulary settings, achieving high intelligibility in an open-vocabulary scenario has remained a significant hurdle, with word error rates (WER) as high as 68%. This limitation hinders the practical application of such technology for communication aids or novel human-computer interfaces. The paper aims to bridge this gap by developing a more robust and flexible model capable of accurately reconstructing speech from subtle facial muscle movements.

Methods

The authors propose an improved transduction model with three key innovations:

  • Direct EMG Feature Learning: Instead of relying on hand-crafted features, the model directly processes raw EMG signals using a stack of residual convolutional neural network (CNN) blocks. This allows the model to learn optimal features for speech synthesis from the EMG data itself, inspired by recent advancements in raw audio processing.
  • Transformer Architecture: The CNN outputs are fed into a bidirectional Transformer encoder. This architecture, known for its effectiveness in capturing long-range dependencies, is employed here with relative position embeddings to better model the temporal nature of speech and EMG signals.
  • Auxiliary Phoneme Prediction: To provide additional guidance during training, an auxiliary task of predicting phoneme labels is introduced. This loss term, inspired by similar work in brain-computer interfaces, helps regularize the model and improve its learned representations, especially in low-data regimes.

The overall pipeline involves the transduction model predicting Mel-frequency cepstral coefficients (MFCCs) from EMG signals, which are then fed into a pre-trained WaveNet vocoder to synthesize the final audio.

Datasets

The model is trained and evaluated on the open-vocabulary dataset from Gaddy and Klein (2020). This dataset comprises 19 hours of synchronized facial EMG and vocalized speech recordings from a single English speaker. For the auxiliary phoneme prediction task, phoneme labels are obtained using the Montreal Forced Aligner (McAuliffe et al., 2017) on the vocalized speech transcripts.

Key Insights and Ideas

  • Learning from Raw Signals is Powerful: Directly learning features from raw EMG signals, rather than relying on pre-defined features, significantly improves the model's ability to capture relevant information for speech synthesis. This highlights the power of end-to-end learning in this domain.
  • Transformers Excel at Temporal Dependencies: The adoption of the Transformer architecture, particularly with relative position embeddings, proves effective in modeling the complex temporal relationships within EMG signals and their correspondence to speech.
  • Auxiliary Tasks Boost Performance: The addition of an auxiliary phoneme prediction loss acts as a valuable regularizer, providing crucial guidance for the model and leading to substantial improvements in intelligibility, especially when data is limited.
  • Automatic Metrics May Underestimate Intelligibility: Human evaluation results show a lower word error rate than automatic transcriptions, suggesting that current automatic metrics might not fully capture the perceived intelligibility of synthesized speech.

Key Results

The proposed model achieves a significant improvement in open-vocabulary silent speech synthesis:

  • Automatic Evaluation: The model reduces the word error rate (WER) from 68.0% (Gaddy and Klein, 2020) to 42.2%, an absolute improvement of 25.8% and a 38% relative error reduction.
  • Ablation Studies: Removing any of the three key contributions (convolutional feature extraction, Transformer architecture, or phoneme loss) leads to a noticeable degradation in performance, underscoring the importance of each component.
  • Human Evaluation: Human evaluators achieved an average WER of 32.3%, further validating the model's improved intelligibility compared to the automatic metric.

Limitations and Future Work

The authors acknowledge that the problem of silent speech synthesis is still far from solved. While their model shows significant progress, the current dataset is limited to a single speaker. Future work could involve expanding the dataset to include multiple speakers and exploring more advanced techniques for handling inter-speaker variability. Additionally, the authors suggest that further investigation into the correlation between phoneme prediction errors and audio synthesis errors could lead to even more targeted improvements. The paper also notes that while the automatic metric is useful for comparison, further research into more human-aligned automatic evaluation metrics would be beneficial.

Summary updated

BibTeX
@inproceedings{gaddy2021improved,
  author = {David Gaddy and Dan Klein},
  title = {{An Improved Model for Voicing Silent Speech}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure