Audio-Visual Speech Inpainting with Deep Learning

Giovanni Morrone, Daniel Michelsanti, Zheng-Hua Tan, Jesper Jensen

Abstract: In this paper, we present a deep-learning-based framework for audio-visual speech inpainting, i.e., the task of restoring the missing parts of an acoustic speech signal from reliable audio context and uncorrupted visual information. Recent work focuses solely on audio-only methods and generally aims at inpainting music signals, which show highly different structure than speech. Instead, we inpaint speech signals with gaps ranging from 100 ms to 1600 ms to investigate the contribution that vision can provide for gaps of different duration. We also experiment with a multi-task learning approach where a phone recognition task is learned together with speech inpainting. Results show that the performance of audio-only speech inpainting approaches degrades rapidly when gaps get large, while the proposed audio-visual approach is able to plausibly restore missing information. In addition, we show that multi-task learning is effective, although the largest contribution to performance comes from vision.

Introduction

Audio signals in real-world environments are frequently corrupted by dropouts, transmission errors, or impulsive noise. "Audio inpainting" aims to fill these gaps using surrounding context. While effective for stationary signals like music, speech is highly dynamic. Traditional audio-only methods struggle when gaps exceed a few hundred milliseconds because the preceding and succeeding audio context loses its predictive power. This research introduces an Audio-Visual Speech Inpainting (AV-SI) framework that leverages visual cues—specifically the speaker's facial movements—to reconstruct missing speech segments for gaps as long as 1.6 seconds.

Methods

The proposed architecture is a speaker-independent neural network designed to fuse multimodal data for temporal reconstruction.

The proposed audio-visual speech inpainting architecture, featuring a BLSTM backbone and a multi-task learning (MTL) subtask for phone recognition.
  1. Feature Extraction: The model processes log-magnitude spectrograms for audio and 68 facial landmark motion vectors for video, tracking the movement of the mouth, nose, and eyes.
  2. Model Architecture: A Bi-directional Long-Short Term Memory (BLSTM) network serves as the core, modeling the temporal dependencies of speech by concatenating audio and video features frame-by-frame.
  3. Multi-Task Learning (MTL): To ensure the generated audio is linguistically meaningful, the network simultaneously performs Phone Recognition. It uses a Connectionist Temporal Classification (CTC) loss to identify phonetic sounds while regressing the missing spectrogram values.
  4. Reconstruction: The network outputs an inpainted magnitude spectrogram. This is merged with the original context, and a Local Weighted Sum (LWS) algorithm estimates the missing phase information to synthesize the final waveform.

Experimental Setup

The study utilized the GRID corpus, a multi-speaker dataset featuring 33 speakers uttering short, structured sentences. The researchers simulated corruption by introducing random gaps, averaging 900ms of missing audio per 3-second clip. To rigorously test the limits of the system, they evaluated performance across fixed gap sizes ranging from 100ms to 1600ms.

Results and Discussion

The integration of visual data proved to be the most critical factor for long-term reconstruction. While audio context is sufficient for tiny "blips," its utility drops sharply for gaps longer than 400ms. In contrast, visual information acts as a stable anchor regardless of the gap duration.

Performance metrics (L1, PER, STOI, PESQ) across different gap sizes, demonstrating that audio-visual models maintain significantly better performance as the gap size increases compared to audio-only models.

Key findings include:

  • Intelligibility and Quality: The AV+MTL model achieved a STOI score of 0.817 and a PESQ of 2.525, significantly outperforming the audio-only baseline (STOI: 0.794, PESQ: 2.458).
  • Gap Resilience: As shown in the gap analysis, the audio-only model's performance plummeted at 1600ms, often "hallucinating" stationary hums. The audio-visual models maintained well-structured spectrograms that sounded like human speech.
  • Phonetic Accuracy: The Phone Error Rate (PER) dropped from 0.228 (Audio-only) to 0.137 (AV+MTL), confirming that multi-task learning helps the model generate more linguistically accurate sounds.

Conclusion

This work demonstrates that visual information is indispensable for speech inpainting over significant durations. While the current study uses a constrained vocabulary (GRID corpus), it establishes a foundation for multimodal speech restoration. Future research will focus on generalizing to unconstrained "in-the-wild" speech and exploring end-to-end waveform generation using GANs or WaveNet to further improve naturalness and phase reconstruction.

Summary updated

Abstract

In this paper, we present a deep-learning-based framework for audio-visual speech inpainting, i.e., the task of restoring the missing parts of an acoustic speech signal from reliable audio context and uncorrupted visual information. Recent work focuses solely on audio-only methods and generally aims at inpainting music signals, which show highly different structure than speech. Instead, we inpaint speech signals with gaps ranging from 100 ms to 1600 ms to investigate the contribution that vision can provide for gaps of different duration. We also experiment with a multi-task learning approach where a phone recognition task is learned together with speech inpainting. Results show that the performance of audio-only speech inpainting approaches degrades rapidly when gaps get large, while the proposed audio-visual approach is able to plausibly restore missing information. In addition, we show that multi-task learning is effective, although the largest contribution to performance comes from vision.
BibTeX
@inproceedings{morrone2021audio,
  author = {Giovanni Morrone and Daniel Michelsanti and Zheng-Hua Tan and Jesper Jensen},
  title = {{Audio-Visual Speech Inpainting with Deep Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413488},
}
Figure