Audio-Visual Speech Inpainting with Deep Learning

Giovanni Morrone, Daniel Michelsanti, Zheng-Hua Tan, Jesper Jensen

TL;DR

Researchers have developed a deep-learning framework that "inpaints" missing segments of speech by looking at the speaker’s face. By combining audio context with facial landmark movements, the system can plausibly restore speech gaps as long as 1.6 seconds—a feat where traditional audio-only methods fail completely.

What Problem Does This Paper Address?

"Inpainting" is a term borrowed from the art world and computer vision, referring to the process of reconstructing lost or deteriorated parts of an image. In the acoustic domain, audio inpainting aims to fill in gaps caused by transmission errors, scratches on a disc, or impulsive noise.

While previous research has successfully inpainted music (which is often repetitive and structured), speech is a much tougher nut to crack. Speech is highly non-stationary and dynamic. If you lose 500ms of a sentence, the surrounding audio context often doesn't contain enough information to guess what was said. This paper addresses the challenge of Speech Inpainting (SI) by leveraging the fact that human speech is inherently multimodal—if we can't hear the speaker, we can often "read" their lips to fill in the blanks.

Methods

The authors propose an Audio-Visual Speech Inpainting (AV-SI) model. Here is how the pipeline works:

  1. Features: The model takes two inputs:
  2. Audio: Log-magnitude spectrograms of the corrupted signal.
  3. Video: 68 facial landmark motion vectors (representing the movement of lips, jaw, and face).
  4. Architecture: The core of the model is a stack of Bi-directional Long Short-Term Memory (BLSTM) units. These are excellent at modeling sequences because they look at both past and future context.
  5. Multi-Task Learning (MTL): To make the model "smarter" about the sounds it's generating, the researchers added a secondary task: Phone Recognition. While the model learns to fill in the spectrogram, it also tries to predict the sequence of phonemes (the smallest units of sound) using a Connectionist Temporal Classification (CTC) loss.
  6. Reconstruction: Since the model only predicts the magnitude of the sound, the researchers used the Local Weighted Sum (LWS) algorithm to estimate the missing phase information and convert the spectrogram back into a listenable waveform.

Datasets

The study used the GRID corpus, a large multi-speaker dataset of audio-visual recordings.

  • Scale: 33 speakers, each uttering 1000 short sentences.
  • Corruption: The researchers artificially "broke" the audio by introducing random gaps. On average, each 3-second clip had 900ms of missing audio, distributed across 1 to 8 gaps.
  • Evaluation: They tested the model on gaps of fixed sizes (100ms, 200ms, 400ms, 800ms, and 1600ms) to see exactly where the audio-only approach breaks down.

Key Insights and Ideas

  • The "Vision Gap": For short gaps (under 200ms), audio-only models perform reasonably well. However, once gaps exceed 400ms, audio-only performance plummets. At 800ms and beyond, the audio context simply doesn't have enough "memory" to reconstruct speech. This is where visual cues become the primary driver of reconstruction.
  • Audio-only "Mumbling": When faced with long gaps, audio-only models tend to generate stationary, low-frequency signals—essentially a generic hum or mumble. In contrast, the audio-visual model generates well-structured spectrograms that correspond to actual speech patterns.
  • Phonetic Awareness Helps: Training the model to recognize phones (MTL) improved the results. By forcing the network to understand what is being said at a linguistic level, it becomes better at reconstructing the sound of those words.
  • Speaker Independence: Unlike some earlier works that required a specific model for every speaker, this framework is speaker-independent, meaning it can generalize to new voices it hasn't seen during training.

Key Results

The researchers compared their AV-SI model against an audio-only baseline using several metrics, including STOI (intelligibility) and PESQ (quality).

  • Intelligibility (STOI): On the longest gaps (1600ms), the audio-only model's STOI score dropped to a dismal 0.3, while the audio-visual model maintained a score above 0.6.
  • Error Rates: The Phone Error Rate (PER) for the audio-visual model was significantly lower (0.137) compared to the audio-only model (0.228), proving that visual data is essential for maintaining the meaning of the speech.
  • Visual Dominance: While Multi-Task Learning provided a nice boost, the jump from "audio-only" to "audio-visual" provided the single largest gain in performance.

Limitations and Future Work

  • Vocabulary Constraints: The GRID corpus uses a very restricted syntax and vocabulary. It remains to be seen how well this generalizes to natural, unconstrained conversation.
  • Phase Reconstruction: The model currently uses a traditional algorithm (LWS) for phase reconstruction. The authors suggest that using a neural vocoder (like WaveNet or HiFi-GAN) could significantly improve the naturalness of the inpainted speech.
  • Inference Speed: While not explicitly discussed as a failure, BLSTMs are computationally expensive; future iterations might explore Transformers or Conformers for faster processing.

Reference: Morrone, G., Michelsanti, D., Tan, Z.-H., & Jensen, J. (2021). Audio-Visual Speech Inpainting with Deep Learning. arXiv:2010.04556v2 [eess.AS].

Summary updated

Abstract

In this paper, we present a deep-learning-based framework for audio-visual speech inpainting, i.e., the task of restoring the missing parts of an acoustic speech signal from reliable audio context and uncorrupted visual information. Recent work focuses solely on audio-only methods and generally aims at inpainting music signals, which show highly different structure than speech. Instead, we inpaint speech signals with gaps ranging from 100 ms to 1600 ms to investigate the contribution that vision can provide for gaps of different duration. We also experiment with a multi-task learning approach where a phone recognition task is learned together with speech inpainting. Results show that the performance of audio-only speech inpainting approaches degrades rapidly when gaps get large, while the proposed audio-visual approach is able to plausibly restore missing information. In addition, we show that multi-task learning is effective, although the largest contribution to performance comes from vision.
BibTeX
@inproceedings{morrone2021audio,
  author = {Giovanni Morrone and Daniel Michelsanti and Zheng-Hua Tan and Jesper Jensen},
  title = {{Audio-Visual Speech Inpainting with Deep Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413488},
}

Loading related content...

Figure