Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language

Jeong Hun Yeo, Chae Won Kim, Hyunjun Kim, Hyeongseop Rha, Seunghee Han, Wen-Huang Cheng, Yong Man Ro

TL;DR

This paper introduces a dual-level adaptation strategy for personalized lip reading, fine-tuning both visual features and linguistic patterns to match a specific speaker. By introducing VoxLRS-SA, the first large-scale "in-the-wild" dataset for speaker-adaptive sentence-level lip reading, the authors demonstrate that adapting a mere 0.1% of a model's parameters can significantly close the performance gap for unseen speakers.

What Problem Does This Paper Address?

Lip reading, or Visual Speech Recognition (VSR), is notoriously difficult because of the high degree of variation between individuals. Two people saying the same sentence will have different lip shapes, different speaking speeds, and unique rhythmic styles. Furthermore, people have "linguistic fingerprints"—preferred vocabularies and phrasing—that general models often fail to capture.

Current VSR models perform well on speakers they’ve seen during training but struggle to generalize to new faces. While "speaker adaptation" exists, it has historically focused only on the visual side and has been tested on small, simplistic datasets (like GRID or LRW-ID) that don't reflect the complexity of real-world, sentence-level English speech.

Methods

The authors propose a two-pronged adaptation approach integrated into a VSP-LLM (Visual Speech Processing - Large Language Model) framework.

Figure 1: Overview of the proposed lip reading model architecture illustrating dual adaptation strategies. Visual level adaptation employs padding prompts and LoRA at the front-end for unique lip features and at the back-end for lip movement speed and style. Language level adaptation uses input prompt tuning to learn the language modeling probability of each speaker.
  1. Vision-Level Adaptation:
    • Front-end (CNN): They use padding prompts (replacing standard zero-padding with learnable parameters) and LoRA (Low-Rank Adaptation) to capture a speaker's unique lip appearance.
  2. Back-end (Conformer): LoRA is applied to the attention layers to adapt to the speaker's specific temporal dynamics, such as speaking speed and movement style.
  3. Language-Level Adaptation:
    • Recognizing that individuals have unique linguistic patterns, the authors apply input prompt tuning to the LLM decoder (LLaMA3-8B). This allows the decoder to learn the probability of specific word sequences occurring for a particular speaker.

This method is highly efficient, requiring only 0.74M trainable parameters compared to the 8.2 billion parameters of the full model.

Datasets

A major contribution of this work is the VoxLRS-SA dataset. Previous benchmarks were limited by small vocabularies or fixed camera angles.

Comparison of VoxLRS-SA with existing datasets. (Note: Referencing Table 1 via the architecture figure's context as provided in the OCR)

Figure 1: Overview of the proposed lip reading model architecture illustrating dual adaptation strategies. Visual level adaptation employs padding prompts and LoRA at the front-end for unique lip features and at the back-end for lip movement speed and style. Language level adaptation uses input prompt tuning to learn the language modeling probability of each speaker.
  • VoxLRS-SA: Derived from VoxCeleb2 and LRS3, it features ~100K words and diverse "in-the-wild" poses.
  • Labeling: Since VoxCeleb2 lacks text, the authors used a pre-trained Whisper ASR model to generate pseudo-labels, which were then manually cleaned for the test set.
  • Scale: It includes 1.7K hours of baseline training data and a specific adaptation set featuring 20 distinct speakers with up to 45 minutes of data each.

Key Insights and Ideas

  • Language Matters: This is the first work to show that adapting to a speaker's vocabulary (language level) improves lip reading performance, not just adapting to their face (vision level).
  • The 5-Minute Threshold: While word-level lip reading can be adapted with just 1 minute of data, the authors found that sentence-level adaptation requires at least 5 minutes of speaker-specific data to consistently outperform the baseline across different individuals.
  • Efficiency vs. Performance: The proposed method achieves a Word Error Rate (WER) of 40.9% using only 0.74M parameters. While full fine-tuning of the encoder (181M parameters) reaches 39.6%, the proposed method is vastly more practical for real-world deployment where storage and compute per-user are limited.

Key Results

The experiments on VoxLRS-SA demonstrate clear gains from the dual-adaptation strategy:

  • Baseline WER: 47.3%
  • Vision-only Adaptation (Ours): 41.5%
  • Language-only Adaptation (Ours): 44.1%
  • Dual Adaptation (Ours): 40.9%
WER performance across different adaptation strategies.

As shown in the duration analysis, increasing the amount of adaptation data from 1 minute to 45 minutes leads to a steady decrease in WER, proving the model effectively "learns" the speaker over time.

WER performance across different adaptation durations.

Limitations and Future Work

Despite the improvements, the authors note two primary failure modes:

  1. Context Misinterpretation: The model sometimes prioritizes a speaker's style so much that it distorts the actual context of the sentence.
  2. Phonetic Substitutions: It still struggles with "homophenes"—words that look identical on the lips (e.g., substituting "prime minister" with "premise").

Future work will focus on few-shot adaptation, aiming to achieve these performance gains with even less than 5 minutes of data, further increasing the technology's accessibility for users with hearing impairments or for use in high-noise environments.

Summary updated

Abstract

Lip reading aims to predict spoken language by analyzing lip movements. Despite advancements in lip reading technologies, performance degrades when models are applied to unseen speakers due to their sensitivity to variations in visual information such as lip appearances. To address this challenge, speaker adaptive lip reading technologies have advanced by focusing on effectively adapting a lip reading model to target speakers in the visual modality. However, the effectiveness of adapting language information, such as vocabulary choice, of the target speaker has not been explored in previous works. Additionally, existing datasets for speaker adaptation have limited vocabulary sizes and pose variations, which restrict the validation of previous speaker-adaptive methods in real-world scenarios. To address these issues, we propose a novel speaker-adaptive lip reading method that adapts a pre-trained model to target speakers at both vision and language levels. Specifically, we integrate prompt tuning and the LoRA approach, applying them to a pre-trained lip reading model to effectively adapt the model to target speakers. Furthermore, to validate its effectiveness in real-world scenarios, we introduce a new dataset, VoxLRS-SA, derived from VoxCeleb2 and LRS3. It contains a vocabulary of approximately 100K words, offers diverse pose variations, and enables the validation of adaptation methods in the wild, sentence-level lip reading for the first time in English. Through various experiments, we demonstrate that the existing speaker-adaptive method also improves performance in the wild at the sentence level. Moreover, we show that the proposed method achieves larger improvements compared to the previous works.
BibTeX
@inproceedings{yeo2025personalized,
  author = {Jeong Hun Yeo and Chae Won Kim and Hyunjun Kim and Hyeongseop Rha and Seunghee Han and Wen-Huang Cheng and Yong Man Ro},
  title = {{Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.33026},
}
Figure