Abstract
Lip reading aims to predict spoken language by analyzing lip movements. Despite advancements in lip reading technologies, performance degrades when models are applied to unseen speakers due to their sensitivity to variations in visual information such as lip appearances. To address this challenge, speaker adaptive lip reading technologies have advanced by focusing on effectively adapting a lip reading model to target speakers in the visual modality. The effectiveness of adapting language information, such as vocabulary choice, of the target speaker has not been explored in the previous works. Moreover, existing datasets for speaker adaptation have limited vocabulary size and pose variations, limiting the validation of previous speaker-adaptive methods in real-world scenarios. To address these issues, we propose a novel speaker-adaptive lip reading method that adapts a pre-trained model to target speakers at both vision and language levels. Specifically, we integrate prompt tuning and the LoRA approach, applying them to a pre-trained lip reading model to effectively adapt the model to target speakers. In addition, to validate its effectiveness in real-world scenarios, we introduce a new dataset, VoxLRS-SA, derived from VoxCeleb2 and LRS3. It contains a vocabulary of approximately 100K words, offers diverse pose variations, and enables the validation of adaptation methods in wild, sentence-level lip reading for the first time. Through various experiments, we demonstrate that the existing speaker-adaptive method also improves performance in the wild at the sentence level. Moreover, with the proposed adaptation method, we show that the proposed method achieves larger improvements when applied to the target speaker, compared to the previous works.
Introduction
Visual Speech Recognition (VSR), or lip reading, faces a significant hurdle: the "unseen speaker" problem. Every individual possesses a unique "visual fingerprint"—variations in lip shape, facial hair, and idiosyncratic movement styles—that causes generic models to struggle. While prior research has focused on adapting visual encoders to these physical traits, it has largely ignored the linguistic dimension. Speakers do not just move their lips differently; they also exhibit distinct lexical patterns and sentence structures. Furthermore, traditional benchmarks like GRID or LRW-ID lack the complexity of real-world speech, featuring limited vocabularies and controlled environments.
Methodology
This work proposes a dual-modality adaptation framework that personalizes a pre-trained VSR model at both the visual and linguistic levels. The architecture utilizes a Conformer-based visual encoder and a frozen LLaMA3-8B decoder.
Vision-Level Adaptation
To account for unique physical appearances, the model employs Padding Prompts in the CNN front-end, replacing standard zero-padding with learned, speaker-specific tokens. Additionally, Low-Rank Adaptation (LoRA) is applied to the attention layers of the visual back-end to capture individual variations in movement speed and style without retraining the entire backbone.
Language-Level Adaptation
The framework treats a speaker's linguistic style as a specific task. Through Input Prompt Tuning, a small set of trainable tokens is concatenated with visual features before being processed by the Large Language Model (LLM). This allows the decoder to adjust its word-prediction probabilities based on the speaker's likely vocabulary and phrasing.
The VoxLRS-SA Dataset
To validate performance in real-world scenarios, the researchers introduced VoxLRS-SA, a benchmark for sentence-level lip reading "in the wild." Derived from VoxCeleb2 and LRS3, it encompasses 1,700 hours of video from approximately 9,000 speakers. Unlike previous datasets, VoxLRS-SA includes diverse head poses, varied backgrounds, and a massive vocabulary of 100,000 words.
Results and Analysis
Experimental results demonstrate that the proposed dual-modality adaptation significantly outperforms baseline and single-modality approaches. The non-adapted baseline recorded a Word Error Rate (WER) of 47.3%. By applying both vision and language adaptation, the WER was reduced to 40.9%.
A key finding is the efficiency of the method: the system achieves these gains by training only 0.74M parameters, representing less than 1% of the original model's trainable parameters. The results indicate that adapting to lexical patterns is a powerful cue; even when visual signals are ambiguous, the LLM can leverage speaker-specific phrasing to improve prediction accuracy.
Conclusion
This research establishes that personalized lip reading requires a holistic approach encompassing both how a person moves and what they say. By introducing the VoxLRS-SA dataset and a parameter-efficient dual-adaptation framework, the authors provide a path toward more robust VSR in real-world applications. Future work will focus on reducing the data requirements for adaptation, moving from minutes of speaker-specific data toward "few-shot" learning in seconds.
Summary updated