Lip2AudSpec: Speech reconstruction from silent lip movements video

Hassan Akbari, Himani Arora, Liangliang Cao, Nima Mesgarani

Abstract

In this study, we propose a deep neural network for reconstructing intelligible speech from silent lip movement videos. We use auditory spectrogram as spectral representation of speech and its corresponding sound generation method resulting in a more natural sounding reconstructed speech. Our proposed network consists of an autoencoder to extract bottleneck features from the auditory spectrogram which is then used as target to our main lip reading network comprising of CNN, LSTM and fully connected layers. Our experiments show that the autoencoder is able to reconstruct the original auditory spectrogram with a 98% correlation and also improves the quality of reconstructed speech from the main lip reading network. Our model, trained jointly on different speakers is able to extract individual speaker characteristics and gives promising results of reconstructing intelligible speech with superior word recognition accuracy.

Introduction

Traditionally, lip reading has been treated as a "video-to-text" classification problem. However, this ignores the rich acoustic style, emotion, and articulation of the speaker. Furthermore, lip reading is inherently difficult because many sounds (phonemes) look identical on the lips (visemes)—for example, distinguishing a "s" from a "z" relies on vocal cord vibration, which is visually hidden.

Recently, researchers have attempted "video-to-speech" generation to synthesize actual audio from silent video. However, previous baselines (like Vid2Speech) struggled to generate natural-sounding audio. They often failed to capture fundamental frequency (pitch) and aperiodicity, resulting in robotic, unnatural voices that lacked speaker identity and had low intelligibility. This paper aims to bridge that gap by generating high-quality, natural-sounding acoustic signals directly from silent video.

Methods

The authors propose a two-part neural network architecture that maps video to an auditory spectrogram (a spectral representation inspired by the human auditory cortex, which yields much higher quality re-synthesis than traditional spectrograms).

Lip2AudSpec Architecture
  1. Network I: The Autoencoder: Auditory spectrograms are high-dimensional (128 frequency bins) and highly correlated, making them difficult for a network to learn directly. The authors pre-train a deep autoencoder to compress these spectrograms into a 32-dimensional bottleneck feature vector. Crucially, they inject additive Gaussian noise at the bottleneck during training, which forces the decoder to become highly robust to variations and noise.
  2. Network II: The Lip Reading Network: The main network takes video slices (processed into 3D tensors including temporal derivatives) and passes them through a 7-layer 3D CNN to extract spatiotemporal features. These features are fed into a 512-unit LSTM to model time dependencies, followed by fully connected layers. The output of this network is the 32-dimensional bottleneck feature, which is then passed to the pre-trained autoencoder's decoder to generate the final auditory spectrogram.
  3. CorrMSE Loss: The authors introduce a custom loss function combining Mean Squared Error (MSE) and Pearson’s correlation coefficient. This ensures the network preserves both the absolute amplitude (MSE) and the structural shape (correlation) of the audio features.

Key Results

The proposed method was evaluated against the state-of-the-art baseline, Vid2Speech, using both algorithmic metrics and human evaluation.

Spectrogram Reconstruction Comparison
  • Quantitative Audio Quality: Lip2AudSpec achieved an average PESQ (Perceptual Evaluation of Speech Quality) of 1.88 (vs. 1.76 for the baseline) and an STMI (intelligibility metric) of 0.80 (vs. 0.52 for the baseline).
  • Frequency Accuracy: The model achieved a 2D Correlation (Corr2D) of 0.88, vastly outperforming the baseline's 0.61, proving its superior ability to reconstruct pitch.
  • Human Evaluation: Human listeners correctly transcribed 55.8% of the words generated by Lip2AudSpec, compared to 50.9% for the baseline. Furthermore, because the model retains pitch, human listeners correctly identified the speaker's gender 85.1% of the time, compared to 43.2% for the baseline.

Conclusion

Lip2AudSpec demonstrates that target compression via a robust autoencoder is key to mapping silent video to high-dimensional audio spectrograms. By recovering fundamental frequencies, the model retains speaker-dependent characteristics, making the audio sound significantly more human. While challenges remain in capturing hidden articulation (like tongue movements) and scaling to open-vocabulary speech, this work represents a significant step toward natural-sounding video-to-speech synthesis.

Summary updated

Abstract

In this study, we propose a deep neural network for reconstructing intelligible speech from silent lip movement videos. We use auditory spectrogram as spectral representation of speech and its corresponding sound generation method resulting in a more natural sounding reconstructed speech. Our proposed network consists of an autoencoder to extract bottleneck features from the auditory spectrogram which is then used as target to our main lip reading network comprising of CNN, LSTM and fully connected layers. Our experiments show that the autoencoder is able to reconstruct the original auditory spectrogram with a 98% correlation and also improves the quality of reconstructed speech from the main lip reading network. Our model, trained jointly on different speakers is able to extract individual speaker characteristics and gives promising results of reconstructing intelligible speech with superior word recognition accuracy.
BibTeX
@inproceedings{akbari2018lip,
  author = {Hassan Akbari and Himani Arora and Liangliang Cao and Nima Mesgarani},
  title = {{Lip2AudSpec: Speech reconstruction from silent lip movements video}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=oCxsR8NBk9F},
}
Figure