LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition

Bowen Hao, Dongliang Zhou, Xiaojie Li, Xingyu Zhang, Liang Xie, Jianlong Wu, Erwei Yin

TL;DR

LipGen is a new framework that boosts Visual Speech Recognition (VSR) performance by using AI-generated synthetic videos to diversify training data. By combining a diffusion-based animation model with a specialized "viseme" (visual speech unit) classification task, it achieves a new state-of-the-art accuracy of 92.8% on the Lip Reading in the Wild (LRW) dataset.

What Problem Does This Paper Address?

Visual Speech Recognition, or "lip reading," is incredibly useful in noisy environments or for silent speech interfaces. However, current AI models are often "brittle." They perform well on standard datasets like LRW, which mostly feature stable, front-facing speakers, but they struggle in the real world where head poses, lighting, and speaker characteristics vary wildly.

The core challenge is data: manually collecting and labeling hundreds of thousands of diverse, high-quality lip-reading videos is prohibitively expensive and time-consuming. Without diverse data, models fail to generalize to the "wild" conditions they are meant to handle.

Methods

The authors propose LipGen, which attacks the problem from two angles: data augmentation and architectural refinement.

  1. Audio-Driven Synthetic Animation: Instead of filming new people, the researchers used AniPortrait (a diffusion-based generative model). They took audio clips from existing datasets and "mapped" them onto diverse facial images from the FFHQ and VGGFace datasets. This allowed them to generate 400,000 synthetic video clips featuring a wide range of ethnicities, poses, and lighting conditions.
  2. Viseme-Guided Auxiliary Task: A "viseme" is the visual equivalent of a phoneme—it’s a group of sounds that look identical on the lips (like 'p', 'b', and 'm'). The authors introduced a secondary branch to the neural network that tries to classify these 18 viseme groups alongside the main word classification task.
  3. Temporal Attention Fusion Module (TAFM): Recognizing that not every frame in a video is equally important for identifying a sound, they developed TAFM. This module uses "prototype learning" to compare video features against a learned "centroid" for each viseme, using an attention mechanism to focus the model on the most relevant frames.

Datasets

  • LRW (Lip Reading in the Wild): The primary benchmark, containing 500 English words spoken by hundreds of different speakers in BBC news broadcasts.
  • FFHQ & VGGFace: Used as the source for "reference images" to create the synthetic speakers.
  • Synthetic Dataset: 400,000 generated clips used to supplement the training of the model.
  • Pose-Augmented LRW: A specialized test set created by the authors to evaluate how well the model handles head rotations (15° to 60°).

Key Insights and Ideas

  • Synthetic Data as a Robustness Tool: While synthetic data only provided a modest boost (0.2%) on the standard, stable LRW test set, it provided a much larger improvement on the "Pose-Augmented" set. This suggests that generative AI is better at teaching models robustness than just raw accuracy.
  • Visemes Provide "Fine-Grained" Supervision: By forcing the model to recognize visemes, the researchers are essentially giving the AI a "cheat sheet" for mouth shapes. This helps the model ignore non-lip-related features (like a speaker's jawline or beard) and focus on the mechanics of speech.
  • The Power of Prototypes: Using a prototype-based attention mechanism (TAFM) allows the model to learn a "representative" version of what a specific lip shape looks like, making it easier to classify noisy or unusual real-world examples.

Key Results

  • New SOTA: LipGen achieved 92.8% accuracy on the LRW dataset, surpassing previous state-of-the-art models like ResNet-18+DC-TCN+cro-TSM (92.4%).
  • Improved Robustness: On the challenging Pose-Augmented LRW test set, the model trained with synthetic data reached 81.8% accuracy, a significant jump over the baseline (79.6%) and the model trained only on real LRW data (80.5%).
  • Ablation Success: The researchers showed that the Viseme Label task (+0.4%) and the TAFM module (+0.4%) each contributed distinct, additive improvements to the final score.

Limitations and Future Work

  • Domain Gap: The authors noted that the performance boost on the standard LRW set from synthetic data alone was relatively small. This suggests a "domain gap" exists where the synthetic videos are still recognizably different from real-world footage to the neural network.
  • Word-Level Focus: This study focused on isolated word recognition. Extending this "viseme-guided" approach to continuous, sentence-level speech recognition remains a logical next step.
  • Hyper-parameter Sensitivity: The ablation study showed that the model is sensitive to the weight ($\lambda$) assigned to the temporal attention module; if the weight is too high or too low, performance drops, suggesting the balance between global averaging and specific attention is delicate.

Summary updated

Abstract

Visual speech recognition (VSR), commonly known as lip reading, has garnered significant attention due to its wide-ranging practical applications. The advent of deep learning techniques and advancements in hardware capabilities have significantly enhanced the performance of lip reading models. Despite these advancements, existing datasets predominantly feature stable video recordings with limited variability in lip movements. This limitation results in models that are highly sensitive to variations encountered in real-world scenarios. To address this issue, we propose a novel framework, LipGen, which aims to improve model robustness by leveraging speech-driven synthetic visual data, thereby mitigating the constraints of current datasets. Additionally, we introduce an auxiliary task that incorporates viseme classification alongside attention mechanisms. This approach facilitates the efficient integration of temporal information, directing the model’s focus toward the relevant segments of speech, thereby enhancing discriminative capabilities. Our method demonstrates superior performance compared to the current state-of-the-art on the lip reading in the wild (LRW) dataset and exhibits even more pronounced advantages under challenging conditions.
BibTeX
@inproceedings{hao2025lipgen,
  author = {Bowen Hao and Dongliang Zhou and Xiaojie Li and Xingyu Zhang and Liang Xie and Jianlong Wu and Erwei Yin},
  title = {{LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=cHOEHRQalK},
}

Loading related content...

Figure