TL;DR
Researchers have developed a new method to reconstruct audible speech from silent "mouthing" movements using ultrasound tongue imaging and lip videos. By generating "pseudo" audio targets through alignment with vocalized recordings and using domain adversarial training to bridge the gap between silent and vocalized speaking modes, the system significantly improves the clarity and naturalness of reconstructed silent speech, cutting word error rates by over 15%.
What Problem Does This Paper Address?
Silent Speech Interfaces (SSIs) aim to convert physical articulatory movements (like the tongue and lips) into acoustic signals. This technology is a lifeline for individuals with dysphonia or those who have undergone laryngectomies, and it has applications in high-noise environments or private communication.
However, reconstructing speech from silent articulation (mouthing words without making sound) is significantly harder than reconstructing it from vocalized articulation. The paper identifies two primary hurdles:
- The Domain Gap: People move their mouths differently when they aren't making sound. Silent articulation tends to be slower, lacks intra-oral pressure, and features less distinct tongue movements compared to speaking aloud.
- The Supervision Gap: In standard supervised learning, you need a "ground truth" audio label for every mouth movement. By definition, silent speech has no corresponding audio, making it impossible to train a model using traditional methods.
Methods
The authors build upon TaLNet, a state-of-the-art encoder-decoder model, and introduce three key innovations to solve the silent speech puzzle:
- Pseudo Target Generation via DTW: To create training labels for silent data, the authors use parallel recordings where the same speaker says the same sentence both silently and aloud. They use Dynamic Time Warping (DTW) to align the vocalized audio to the silent articulatory movements. This "stretches" the audio to match the timing of the silent mouth movements, creating a "pseudo target" for the model to learn from.
- Domain Adversarial Training (DAT): To ensure the model doesn't get confused by the differences between silent and vocalized modes, they implement a domain discriminator. This component tries to guess whether a set of articulatory features comes from a silent or vocalized recording. Meanwhile, the encoder is trained to "fool" the discriminator using a Gradient Reversal Layer (GRL). This forces the encoder to extract robust, "domain-invariant" features that represent the intent of the speech regardless of the speaking mode.
- Iterative Training Strategy: The process is cyclical. A better encoder produces better alignments for pseudo targets, and better pseudo targets lead to a better encoder. The authors repeat the alignment and training steps to progressively refine the model's performance.
Datasets
The study utilizes the Tongue and Lip (TaL) dataset, which contains synchronized audio, ultrasound tongue images, and optical lip videos from 81 native English speakers. The dataset is unique because it includes parallel recordings of the same linguistic content in both vocalized and silent modes.
Key Insights and Ideas
- Silent Speech is its own "Language": The authors treat silent and vocalized speech as two different "domains" (similar to how a model might treat photos vs. sketches). This allows them to use transfer learning and adversarial techniques typically reserved for computer vision.
- The "Reliability" Filter: Not all DTW alignments are perfect. The authors implemented a cost threshold to discard "unreliable" pseudo targets, ensuring the model only learns from high-quality alignments.
- Cross-Mode Benefits: Interestingly, training on silent data using pseudo targets actually improved the model's performance on vocalized speech as well. This suggests that the pseudo targets act as a form of data augmentation, helping the model generalize better overall.
Key Results
The proposed method was compared against a baseline TaLNet model using both objective metrics and subjective listening tests:
- Intelligibility (WER): In silent mode, the Word Error Rate (WER) dropped from 59.96% to 43.11% (a relative improvement of over 15%). Even in vocalized mode, the WER improved from 26.89% to 17.31%.
- Naturalness (MOS): Human listeners rated the reconstructed silent speech significantly higher. The Mean Opinion Score (MOS) rose from 2.99 to 3.33 on a 5-point scale.
- Acoustic Accuracy: The Mel-Cepstral Distortion (MCD)—a measure of how much the reconstructed signal deviates from the target—decreased significantly, indicating more accurate spectral reconstruction.
Limitations and Future Work
While the improvements are substantial, the authors acknowledge that a significant gap still exists between the quality of reconstructed silent speech and natural vocalized speech.
One major limitation is that the model currently relies on parallel data (the same speaker saying the same thing twice) to generate pseudo targets. Future work will likely focus on how to bridge the domain gap without requiring such strictly paired recordings. Additionally, the authors aim to further investigate the intrinsic physical differences in articulation between the two modes to create even more specialized architectures.
Summary updated