LipSound: Neural Mel-Spectrogram Reconstruction for Lip Reading

Leyuan Qu, Cornelius Weber, Stefan Wermter

Abstract

Visual speech recognition (VSR), commonly known as lip reading, is a difficult task due to the inherent ambiguity of visual speech signals. This paper presents LipSound, a novel deep learning architecture that bridges the gap between sight and sound by first reconstructing a mel-spectrogram from silent video before transcribing it into text. By leveraging an intermediate acoustic representation, LipSound effectively resolves viseme ambiguities and captures critical temporal context. Our approach achieves state-of-the-art performance on the GRID benchmark with a 2.525% Word Error Rate (WER), significantly outperforming direct video-to-text models.

Introduction: The Challenge of Visual Speech

Lip reading is fundamentally constrained by the "impoverished" nature of the visual signal compared to audio. Two primary hurdles hinder traditional VSR systems:

  1. Missing Information: Invisible speech components, such as vocal cord vibration (voicing), are absent from the video.
  2. The Viseme Problem: Many distinct phonemes (sounds) appear identical on the lips (e.g., /p/ and /b/), making them indistinguishable from static frames.

LipSound addresses these issues by forcing the model to "hallucinate" the missing acoustic information, using temporal context to resolve visual ambiguities.

The LipSound Architecture

The LipSound system employs a two-stage pipeline designed to translate visual sequences into text via an acoustic bridge.

The LipSound architecture: a two-stage pipeline consisting of a mel-spectrogram generator (front-end) and a DeepSpeech 2-based lip reading system (back-end).

1. Front-end: Mel-spectrogram Generator

Inspired by the Tacotron2 architecture, the front-end uses an encoder-decoder structure with location-sensitive attention.

  • Encoder: Processes mouth image sequences (compressed via PCA) through 1D convolutional layers and a Bi-LSTM to extract temporal features.
  • Decoder: Predicts the mel-spectrogram frame-by-frame. During training, teacher-forcing with ground-truth audio is utilized to guide the mapping.
  • Post-Net: A 5-layer CNN refines the generated spectrogram to enhance its quality.

2. Back-end: Lip Reading System

The back-end is a speech-to-text engine based on DeepSpeech 2. It processes the generated mel-spectrogram through 2D convolutions and Bi-GRU layers, using Connectionist Temporal Classification (CTC) loss to produce the final character sequence.

Key Insights: Acoustic Hallucination and Context

The core innovation of LipSound is the use of an intermediate acoustic representation. This "acoustic hallucination" acts as a powerful feature extractor. While a /p/ and /b/ may look identical in a single frame, their acoustic transitions into subsequent vowels differ. By reconstructing the full spectrogram, the model captures these subtle temporal dependencies. Furthermore, this approach allows the front-end to be trained on large volumes of unlabeled video-audio pairs, bypassing the need for expensive text transcripts.

Experimental Results

LipSound was evaluated on the GRID dataset, which contains 34,000 videos of speakers reciting structured sentences.

Qualitative comparison of generated mel-spectrograms. (a) shows successful reconstructions for different sentences. (b) highlights the visual ambiguity between similar sounding letters like 'U' vs 'O' and 'R' vs 'E'.

The model achieved a 0.843% Character Error Rate (CER) and a 2.525% Word Error Rate (WER), setting a new state-of-the-art on this benchmark. It significantly outperformed previous models such as LipNet (1.9% CER) and LCANet (1.3% CER). Qualitatively, the generated spectrograms were found to be intelligible and even preserved speaker-specific vocal characteristics when converted back to audio.

Limitations and Future Work

Despite its success, LipSound faces challenges with isolated letter ambiguity (e.g., 'O' vs 'U'), where linguistic context is insufficient for the decoder to differentiate between visually similar inputs. Additionally, the current model is optimized for the restricted grammar of the GRID dataset. Future research will focus on scaling to spontaneous speech, developing speaker-independent models, and exploring multi-modal fusion for robust speech recognition in noisy environments.

Summary updated

Abstract

Lip reading, also known as visual speech recognition, has recently received considerable attention. Although advanced feature engineering and powerful deep neural network architectures have been proposed for this task, the performance still cannot be competitive with speech recognition tasks using the audio modality as input. This is mainly because compared with audio, visual features carry less information relevant to word recognition. For example, the voiced sound made while the vocal cords vibrate can be represented by audio but is not reflected by mouth or lip movement. In this paper, we map the sequence of mouth movement images directly to mel-spectrogram to reconstruct the speech relevant information. Our proposed architecture consists of two components: (a) the mel-spectrogram reconstruction front-end which includes an encoder-decoder architecture with attention mechanism to predict mel-spectrogram from videos; (b) the lip reading back-end consisting of convolutional layers, bi-directional gated recurrent units, and connectionist temporal classification loss, which consumes the generated mel-spectrogram representation to predict text transcriptions. The speaker-dependent evaluation results demonstrate that our proposed model not only generates quality mel-spectrograms but also outperforms state-of-the-art models on the GRID benchmark lip reading dataset, with 0.843% character error rate and 2.525% word error rate.
BibTeX
@inproceedings{qu2019lipsound,
  author = {Leyuan Qu and Cornelius Weber and Stefan Wermter},
  title = {{LipSound: Neural Mel-Spectrogram Reconstruction for Lip Reading}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1393},
}
Figure