Speech Reconstruction from Silent Tongue and Lip Articulation by Pseudo Target Generation and Domain Adversarial Training

Rui-Chen Zheng, Yang Ai, Zhen-Hua LingICASSP 2023 · DOI · OpenReview

TL;DR

Researchers have developed a new method to reconstruct audible speech from silent "mouthing" movements using ultrasound tongue imaging and lip videos. By generating "pseudo" audio targets through alignment with vocalized recordings and using domain adversarial training to bridge the gap between silent and vocalized speaking modes, the system significantly improves the clarity and naturalness of reconstructed silent speech, cutting word error rates by over 15%.

What Problem Does This Paper Address?

Silent Speech Interfaces (SSIs) aim to convert physical articulatory movements (like the tongue and lips) into acoustic signals. This technology is a lifeline for individuals with dysphonia or those who have undergone laryngectomies, and it has applications in high-noise environments or private communication.

However, reconstructing speech from silent articulation (mouthing words without making sound) is significantly harder than reconstructing it from vocalized articulation. The paper identifies two primary hurdles:

  1. The Domain Gap: People move their mouths differently when they aren't making sound. Silent articulation tends to be slower, lacks intra-oral pressure, and features less distinct tongue movements compared to speaking aloud.
  2. The Supervision Gap: In standard supervised learning, you need a "ground truth" audio label for every mouth movement. By definition, silent speech has no corresponding audio, making it impossible to train a model using traditional methods.

Methods

The authors build upon TaLNet, a state-of-the-art encoder-decoder model, and introduce three key innovations to solve the silent speech puzzle:

  • Pseudo Target Generation via DTW: To create training labels for silent data, the authors use parallel recordings where the same speaker says the same sentence both silently and aloud. They use Dynamic Time Warping (DTW) to align the vocalized audio to the silent articulatory movements. This "stretches" the audio to match the timing of the silent mouth movements, creating a "pseudo target" for the model to learn from.
  • Domain Adversarial Training (DAT): To ensure the model doesn't get confused by the differences between silent and vocalized modes, they implement a domain discriminator. This component tries to guess whether a set of articulatory features comes from a silent or vocalized recording. Meanwhile, the encoder is trained to "fool" the discriminator using a Gradient Reversal Layer (GRL). This forces the encoder to extract robust, "domain-invariant" features that represent the intent of the speech regardless of the speaking mode.
  • Iterative Training Strategy: The process is cyclical. A better encoder produces better alignments for pseudo targets, and better pseudo targets lead to a better encoder. The authors repeat the alignment and training steps to progressively refine the model's performance.

Datasets

The study utilizes the Tongue and Lip (TaL) dataset, which contains synchronized audio, ultrasound tongue images, and optical lip videos from 81 native English speakers. The dataset is unique because it includes parallel recordings of the same linguistic content in both vocalized and silent modes.

Key Insights and Ideas

  • Silent Speech is its own "Language": The authors treat silent and vocalized speech as two different "domains" (similar to how a model might treat photos vs. sketches). This allows them to use transfer learning and adversarial techniques typically reserved for computer vision.
  • The "Reliability" Filter: Not all DTW alignments are perfect. The authors implemented a cost threshold to discard "unreliable" pseudo targets, ensuring the model only learns from high-quality alignments.
  • Cross-Mode Benefits: Interestingly, training on silent data using pseudo targets actually improved the model's performance on vocalized speech as well. This suggests that the pseudo targets act as a form of data augmentation, helping the model generalize better overall.

Key Results

The proposed method was compared against a baseline TaLNet model using both objective metrics and subjective listening tests:

  • Intelligibility (WER): In silent mode, the Word Error Rate (WER) dropped from 59.96% to 43.11% (a relative improvement of over 15%). Even in vocalized mode, the WER improved from 26.89% to 17.31%.
  • Naturalness (MOS): Human listeners rated the reconstructed silent speech significantly higher. The Mean Opinion Score (MOS) rose from 2.99 to 3.33 on a 5-point scale.
  • Acoustic Accuracy: The Mel-Cepstral Distortion (MCD)—a measure of how much the reconstructed signal deviates from the target—decreased significantly, indicating more accurate spectral reconstruction.

Limitations and Future Work

While the improvements are substantial, the authors acknowledge that a significant gap still exists between the quality of reconstructed silent speech and natural vocalized speech.

One major limitation is that the model currently relies on parallel data (the same speaker saying the same thing twice) to generate pseudo targets. Future work will likely focus on how to bridge the domain gap without requiring such strictly paired recordings. Additionally, the authors aim to further investigate the intrinsic physical differences in articulation between the two modes to create even more specialized architectures.

Summary updated

Abstract

This paper studies the task of speech reconstruction from ultrasound tongue images and optical lip videos recorded in a silent speaking mode, where people only activate their intra-oral and extra-oral articulators without producing sound. This task falls under the umbrella of articulatory-to-acoustic conversion, and may also be refered to as a silent speech interface. We propose to employ a method built on pseudo target generation and domain adversarial training with an iterative training strategy to improve the intelligibility and naturalness of the speech recovered from silent tongue and lip articulation. Experiments show that our proposed method significantly improves the intelligibility and naturalness of the reconstructed speech in silent speaking mode compared to the baseline TaLNet model. When using an automatic speech recognition (ASR) model to measure intelligibility, the word error rate (WER) of our proposed method decreases by over 15% compared to the baseline. In addition, our proposed method also outperforms the baseline on the intelligibility of the speech reconstructed in vocalized articulating mode, reducing the WER by approximately 10%.
BibTeX
@inproceedings{zheng2023speech,
  author = {Rui-Chen Zheng and Yang Ai and Zhen-Hua Ling},
  title = {{Speech Reconstruction from Silent Tongue and Lip Articulation by Pseudo Target Generation and Domain Adversarial Training}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.48550/arxiv.2304.05574},
}
Figure