TaLNet: Voice Reconstruction from Tongue and Lip Articulation with Transfer Learning from Text-to-Speech Synthesis

Jing-Xuan Zhang, Korin Richmond, Zhen-Hua Ling, Lirong Dai

Introduction

TaLNet is a novel deep learning architecture designed to reconstruct high-quality speech from ultrasound tongue images and optical lip videos. By leveraging transfer learning from a pre-trained text-to-speech (TTS) model, researchers have addressed the chronic shortage of articulatory data, achieving a 30% relative reduction in Word Error Rate (WER) compared to previous state-of-the-art methods. This technology holds significant promise for developing "Silent Speech Interfaces" (SSIs) for patients who have lost their voices or for communication in high-noise environments.

Addressing the Challenges of Articulatory-to-Acoustic Conversion

The field of articulatory-to-acoustic conversion faces three primary hurdles: data scarcity, model complexity, and significant speaker variation. While standard speech datasets are vast, synchronized ultrasound and lip video data are extremely limited. Furthermore, individual differences in "tongue signatures" make it difficult for multi-speaker models to generalize across different users.

Methods

The proposed TaLNet architecture is an encoder-decoder model designed to bridge the gap between physical movement and digital sound.

The TaLNet architecture featuring dual encoders for tongue and lip data and an acoustic decoder derived from TTS pre-training.

Model Components

  • The Encoders: The model employs two separate encoders based on 3D Convolutional Neural Networks (3D CNNs) to process ultrasound tongue video and optical lip video. To account for speaker variation, the model incorporates the mean and standard deviation of a speaker's images as a "speaker code."
  • The Decoder: Rather than training from scratch, the researchers utilized a transfer learning strategy. They "transplanted" a decoder from a standard Tacotron 2 model pre-trained on a massive audio-only dataset (LibriTTS).
  • Forced Attention: Unlike standard TTS, where alignment must be learned, articulatory movements are naturally synchronized with sound. TaLNet replaces standard "soft" attention with forced attention, ensuring a 1:1 temporal mapping between input and output.

Training Strategy

The model is trained through a rigorous four-stage process to maximize the benefits of transfer learning and joint optimization.

The four-stage training pipeline, moving from TTS pre-training to multi-speaker and finally single-speaker fine-tuning.

Datasets

  • Ultrasuite: A multi-speaker dataset containing ultrasound tongue imaging, lip video, and synchronized audio from 75 speakers (over 10,000 utterances).
  • LibriTTS: A large-scale corpus of English speech (460 hours) used for the initial Stage 1 pre-training of the Tacotron 2 decoder.

Key Insights and Results

TTS as a Foundation

A core innovation of TaLNet is the realization that articulatory-to-acoustic conversion and TTS share the same goal: generating natural-sounding speech. Pre-training on TTS allows the model to learn the fundamental characteristics of human speech before processing any articulatory data.

Performance Metrics

TaLNet significantly outperformed traditional Deep Neural Network (DNN) and Bidirectional LSTM (BLSTM) baselines:

  • Intelligibility: In automated speech recognition (ASR) tests, TaLNet achieved a Word Error Rate (WER) of 36.5%, a substantial improvement over the BLSTM baseline of 53.6%.
  • Naturalness: MUSHRA listening tests confirmed that TaLNet was rated significantly higher in both naturalness and intelligibility by human listeners.
  • Data Efficiency: The transfer learning strategy proved robust, allowing the model to maintain high performance even with a reduced number of training speakers.

Limitations and Future Work

While TaLNet performs well on audible speech, performance declines during "silent" articulation, where speakers often lack the auditory feedback to maintain consistent pacing. Future research will focus on real-time implementation to provide auditory feedback to users and improving computer vision preprocessing to handle variations in imaging quality, such as camera tilt or unclear ultrasound edges.

Summary updated

Abstract

This paper presents TaLNet, a model for voice reconstruction with ultrasound tongue and optical lip videos as inputs. TaLNet is based on an encoder-decoder architecture. Separate encoders are dedicated to processing the tongue and lip data streams respectively. The decoder predicts acoustic features conditioned on encoder outputs and speaker codes.To mitigate for having only relatively small amounts of dual articulatory-acoustic data available for training, and since our task here shares with text-to-speech (TTS) the common goal of speech generation, we propose a novel transfer learning strategy to exploit the much larger amounts of acoustic-only data available to train TTS models. For this, a Tacotron 2 TTS model is first trained, and then the parameters of its decoder are transferred to the TaLNet decoder. We have evaluated our approach on an unconstrained multi-speaker voice recovery task. Our results show the effectiveness of both the proposed model and the transfer learning strategy. Speech reconstructed using our proposed method significantly outperformed all baselines (DNN, BLSTM and without transfer learning) in terms of both naturalness and intelligibility. When using an ASR model decoding the recovery speech, the WER of our proposed method is relatively reduced over 30% compared to baselines.
BibTeX
@inproceedings{zhang2021talnet,
  author = {Jing-Xuan Zhang and Korin Richmond and Zhen-Hua Ling and Lirong Dai},
  title = {{TaLNet: Voice Reconstruction from Tongue and Lip Articulation with Transfer Learning from Text-to-Speech Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i16.17693},
}
Figure