Abstract
Visual speech recognition (VSR), commonly known as lip reading, has garnered significant attention due to its wide-ranging practical applications. The advent of deep learning techniques and advancements in hardware capabilities have significantly enhanced the performance of lip reading models. Despite these advancements, existing datasets predominantly feature stable video recordings with limited variability in lip movements. This limitation results in models that are highly sensitive to variations encountered in real-world scenarios. To address this issue, we propose a novel framework, LipGen, which aims to improve model robustness by leveraging speech-driven synthetic visual data, thereby mitigating the constraints of current datasets. Additionally, we introduce an auxiliary task that incorporates viseme classification alongside attention mechanisms. This approach facilitates the efficient integration of temporal information, directing the model's focus toward the relevant segments of speech, thereby enhancing discriminative capabilities. Our method demonstrates superior performance compared to the current state-of-the-art on the lip reading in the wild (LRW) dataset and exhibits even more pronounced advantages under challenging conditions.
Introduction
Visual Speech Recognition (VSR) is a critical technology for enabling communication in noisy environments and developing silent speech interfaces. However, current AI models are often "brittle." While they perform well on standard benchmarks like the Lip Reading in the Wild (LRW) dataset—which primarily features stable, front-facing speakers—they often fail in real-world conditions where head poses, lighting, and speaker characteristics vary significantly.
The core challenge lies in the data. Manually collecting and labeling hundreds of thousands of diverse, high-quality lip-reading videos is prohibitively expensive. Without diverse training data, models fail to generalize to the "wild" conditions they are intended to handle. LipGen addresses this by utilizing generative AI to create diverse synthetic training samples and refining the model architecture to focus on essential visual speech units.
The LipGen Framework
The LipGen framework attacks the robustness problem from two angles: data augmentation through synthetic animation and architectural refinement via viseme guidance.
Audio-Driven Synthetic Animation
To diversify the training set without manual filming, we utilize AniPortrait, a diffusion-based generative model. By mapping audio clips from existing datasets onto diverse facial images from the FFHQ and VGGFace datasets, we generated 400,000 synthetic video clips. These clips encompass a wide range of ethnicities, poses, and lighting conditions, providing the model with the variety needed to learn robust features.
Viseme-Guided Auxiliary Task
A "viseme" is the visual equivalent of a phoneme—a group of sounds that appear identical on the lips (e.g., 'p', 'b', and 'm'). LipGen introduces an auxiliary branch to the neural network dedicated to classifying 18 viseme groups alongside the main word classification task. This provides fine-grained supervision, helping the model ignore non-speech features like jawlines or facial hair and focus on the mechanics of speech.
Temporal Attention Fusion Module (TAFM)
Recognizing that not every frame in a video is equally important for identifying a sound, we developed the Temporal Attention Fusion Module (TAFM). This module uses prototype learning to compare video features against a learned "centroid" for each viseme, using an attention mechanism to focus the model on the most relevant frames.
Experimental Results
We evaluated LipGen on the LRW benchmark and a specialized "Pose-Augmented" test set created to measure robustness to head rotations (15° to 60°).
As shown in the table above, LipGen achieves a new state-of-the-art accuracy of 92.8% on the standard LRW dataset, surpassing previous models like ResNet-18+DC-TCN+cro-TSM.
Beyond raw accuracy, the model demonstrates significant gains in robustness. On the challenging Pose-Augmented LRW test set, the model trained with synthetic data reached 81.8% accuracy, a notable improvement over the baseline (79.6%) and the model trained only on real LRW data (80.5%). Ablation studies confirmed that the viseme classification task (+0.4%) and the TAFM module (+0.4%) each contributed distinct, additive improvements to the final performance.
Conclusion
LipGen demonstrates that generative AI can be a powerful tool for enhancing the robustness of VSR models. By supplementing real-world data with diverse synthetic samples and providing fine-grained supervision through viseme classification, we can build lip-reading systems that perform reliably outside of controlled environments. Future work will focus on bridging the remaining domain gap between synthetic and real videos and extending this viseme-guided approach to continuous, sentence-level speech recognition.
Summary updated