Abstract
Visual speech recognition (VSR), commonly known as lip reading, is a difficult task due to the inherent ambiguity of visual speech signals. This paper presents LipSound, a novel deep learning architecture that bridges the gap between sight and sound by first reconstructing a mel-spectrogram from silent video before transcribing it into text. By leveraging an intermediate acoustic representation, LipSound effectively resolves viseme ambiguities and captures critical temporal context. Our approach achieves state-of-the-art performance on the GRID benchmark with a 2.525% Word Error Rate (WER), significantly outperforming direct video-to-text models.
Introduction: The Challenge of Visual Speech
Lip reading is fundamentally constrained by the "impoverished" nature of the visual signal compared to audio. Two primary hurdles hinder traditional VSR systems:
- Missing Information: Invisible speech components, such as vocal cord vibration (voicing), are absent from the video.
- The Viseme Problem: Many distinct phonemes (sounds) appear identical on the lips (e.g., /p/ and /b/), making them indistinguishable from static frames.
LipSound addresses these issues by forcing the model to "hallucinate" the missing acoustic information, using temporal context to resolve visual ambiguities.
The LipSound Architecture
The LipSound system employs a two-stage pipeline designed to translate visual sequences into text via an acoustic bridge.
1. Front-end: Mel-spectrogram Generator
Inspired by the Tacotron2 architecture, the front-end uses an encoder-decoder structure with location-sensitive attention.
- Encoder: Processes mouth image sequences (compressed via PCA) through 1D convolutional layers and a Bi-LSTM to extract temporal features.
- Decoder: Predicts the mel-spectrogram frame-by-frame. During training, teacher-forcing with ground-truth audio is utilized to guide the mapping.
- Post-Net: A 5-layer CNN refines the generated spectrogram to enhance its quality.
2. Back-end: Lip Reading System
The back-end is a speech-to-text engine based on DeepSpeech 2. It processes the generated mel-spectrogram through 2D convolutions and Bi-GRU layers, using Connectionist Temporal Classification (CTC) loss to produce the final character sequence.
Key Insights: Acoustic Hallucination and Context
The core innovation of LipSound is the use of an intermediate acoustic representation. This "acoustic hallucination" acts as a powerful feature extractor. While a /p/ and /b/ may look identical in a single frame, their acoustic transitions into subsequent vowels differ. By reconstructing the full spectrogram, the model captures these subtle temporal dependencies. Furthermore, this approach allows the front-end to be trained on large volumes of unlabeled video-audio pairs, bypassing the need for expensive text transcripts.
Experimental Results
LipSound was evaluated on the GRID dataset, which contains 34,000 videos of speakers reciting structured sentences.
The model achieved a 0.843% Character Error Rate (CER) and a 2.525% Word Error Rate (WER), setting a new state-of-the-art on this benchmark. It significantly outperformed previous models such as LipNet (1.9% CER) and LCANet (1.3% CER). Qualitatively, the generated spectrograms were found to be intelligible and even preserved speaker-specific vocal characteristics when converted back to audio.
Limitations and Future Work
Despite its success, LipSound faces challenges with isolated letter ambiguity (e.g., 'O' vs 'U'), where linguistic context is insufficient for the decoder to differentiate between visually similar inputs. Additionally, the current model is optimized for the restricted grammar of the GRID dataset. Future research will focus on scaling to spontaneous speech, developing speaker-independent models, and exploring multi-modal fusion for robust speech recognition in noisy environments.
Summary updated