TL;DR
This paper introduces a dual-level adaptation strategy for personalized lip reading, fine-tuning both visual features and linguistic patterns to match a specific speaker. By introducing VoxLRS-SA, the first large-scale "in-the-wild" dataset for speaker-adaptive sentence-level lip reading, the authors demonstrate that adapting a mere 0.1% of a model's parameters can significantly close the performance gap for unseen speakers.
What Problem Does This Paper Address?
Lip reading, or Visual Speech Recognition (VSR), is notoriously difficult because of the high degree of variation between individuals. Two people saying the same sentence will have different lip shapes, different speaking speeds, and unique rhythmic styles. Furthermore, people have "linguistic fingerprints"—preferred vocabularies and phrasing—that general models often fail to capture.
Current VSR models perform well on speakers they’ve seen during training but struggle to generalize to new faces. While "speaker adaptation" exists, it has historically focused only on the visual side and has been tested on small, simplistic datasets (like GRID or LRW-ID) that don't reflect the complexity of real-world, sentence-level English speech.
Methods
The authors propose a two-pronged adaptation approach integrated into a VSP-LLM (Visual Speech Processing - Large Language Model) framework.
- Vision-Level Adaptation:
- Front-end (CNN): They use padding prompts (replacing standard zero-padding with learnable parameters) and LoRA (Low-Rank Adaptation) to capture a speaker's unique lip appearance.
- Back-end (Conformer): LoRA is applied to the attention layers to adapt to the speaker's specific temporal dynamics, such as speaking speed and movement style.
- Language-Level Adaptation:
- Recognizing that individuals have unique linguistic patterns, the authors apply input prompt tuning to the LLM decoder (LLaMA3-8B). This allows the decoder to learn the probability of specific word sequences occurring for a particular speaker.
This method is highly efficient, requiring only 0.74M trainable parameters compared to the 8.2 billion parameters of the full model.
Datasets
A major contribution of this work is the VoxLRS-SA dataset. Previous benchmarks were limited by small vocabularies or fixed camera angles.
Comparison of VoxLRS-SA with existing datasets. (Note: Referencing Table 1 via the architecture figure's context as provided in the OCR)
- VoxLRS-SA: Derived from VoxCeleb2 and LRS3, it features ~100K words and diverse "in-the-wild" poses.
- Labeling: Since VoxCeleb2 lacks text, the authors used a pre-trained Whisper ASR model to generate pseudo-labels, which were then manually cleaned for the test set.
- Scale: It includes 1.7K hours of baseline training data and a specific adaptation set featuring 20 distinct speakers with up to 45 minutes of data each.
Key Insights and Ideas
- Language Matters: This is the first work to show that adapting to a speaker's vocabulary (language level) improves lip reading performance, not just adapting to their face (vision level).
- The 5-Minute Threshold: While word-level lip reading can be adapted with just 1 minute of data, the authors found that sentence-level adaptation requires at least 5 minutes of speaker-specific data to consistently outperform the baseline across different individuals.
- Efficiency vs. Performance: The proposed method achieves a Word Error Rate (WER) of 40.9% using only 0.74M parameters. While full fine-tuning of the encoder (181M parameters) reaches 39.6%, the proposed method is vastly more practical for real-world deployment where storage and compute per-user are limited.
Key Results
The experiments on VoxLRS-SA demonstrate clear gains from the dual-adaptation strategy:
- Baseline WER: 47.3%
- Vision-only Adaptation (Ours): 41.5%
- Language-only Adaptation (Ours): 44.1%
- Dual Adaptation (Ours): 40.9%
As shown in the duration analysis, increasing the amount of adaptation data from 1 minute to 45 minutes leads to a steady decrease in WER, proving the model effectively "learns" the speaker over time.
Limitations and Future Work
Despite the improvements, the authors note two primary failure modes:
- Context Misinterpretation: The model sometimes prioritizes a speaker's style so much that it distorts the actual context of the sentence.
- Phonetic Substitutions: It still struggles with "homophenes"—words that look identical on the lips (e.g., substituting "prime minister" with "premise").
Future work will focus on few-shot adaptation, aiming to achieve these performance gains with even less than 5 minutes of data, further increasing the technology's accessibility for users with hearing impairments or for use in high-noise environments.
Summary updated