Introduction
TaLNet is a novel deep learning architecture designed to reconstruct high-quality speech from ultrasound tongue images and optical lip videos. By leveraging transfer learning from a pre-trained text-to-speech (TTS) model, researchers have addressed the chronic shortage of articulatory data, achieving a 30% relative reduction in Word Error Rate (WER) compared to previous state-of-the-art methods. This technology holds significant promise for developing "Silent Speech Interfaces" (SSIs) for patients who have lost their voices or for communication in high-noise environments.
Addressing the Challenges of Articulatory-to-Acoustic Conversion
The field of articulatory-to-acoustic conversion faces three primary hurdles: data scarcity, model complexity, and significant speaker variation. While standard speech datasets are vast, synchronized ultrasound and lip video data are extremely limited. Furthermore, individual differences in "tongue signatures" make it difficult for multi-speaker models to generalize across different users.
Methods
The proposed TaLNet architecture is an encoder-decoder model designed to bridge the gap between physical movement and digital sound.
Model Components
- The Encoders: The model employs two separate encoders based on 3D Convolutional Neural Networks (3D CNNs) to process ultrasound tongue video and optical lip video. To account for speaker variation, the model incorporates the mean and standard deviation of a speaker's images as a "speaker code."
- The Decoder: Rather than training from scratch, the researchers utilized a transfer learning strategy. They "transplanted" a decoder from a standard Tacotron 2 model pre-trained on a massive audio-only dataset (LibriTTS).
- Forced Attention: Unlike standard TTS, where alignment must be learned, articulatory movements are naturally synchronized with sound. TaLNet replaces standard "soft" attention with forced attention, ensuring a 1:1 temporal mapping between input and output.
Training Strategy
The model is trained through a rigorous four-stage process to maximize the benefits of transfer learning and joint optimization.
Datasets
- Ultrasuite: A multi-speaker dataset containing ultrasound tongue imaging, lip video, and synchronized audio from 75 speakers (over 10,000 utterances).
- LibriTTS: A large-scale corpus of English speech (460 hours) used for the initial Stage 1 pre-training of the Tacotron 2 decoder.
Key Insights and Results
TTS as a Foundation
A core innovation of TaLNet is the realization that articulatory-to-acoustic conversion and TTS share the same goal: generating natural-sounding speech. Pre-training on TTS allows the model to learn the fundamental characteristics of human speech before processing any articulatory data.
Performance Metrics
TaLNet significantly outperformed traditional Deep Neural Network (DNN) and Bidirectional LSTM (BLSTM) baselines:
- Intelligibility: In automated speech recognition (ASR) tests, TaLNet achieved a Word Error Rate (WER) of 36.5%, a substantial improvement over the BLSTM baseline of 53.6%.
- Naturalness: MUSHRA listening tests confirmed that TaLNet was rated significantly higher in both naturalness and intelligibility by human listeners.
- Data Efficiency: The transfer learning strategy proved robust, allowing the model to maintain high performance even with a reduced number of training speakers.
Limitations and Future Work
While TaLNet performs well on audible speech, performance declines during "silent" articulation, where speakers often lack the auditory feedback to maintain consistent pacing. Future research will focus on real-time implementation to provide auditory feedback to users and improving computer vision preprocessing to handle variations in imaging quality, such as camera tilt or unclear ultrasound edges.
Summary updated