VALLR: Visual ASR Language Model for Lip Reading

Marshall Thomas, Edward Fish, Richard Bowden

Abstract

Lip Reading, or Visual Automatic Speech Recognition (V-ASR), is a complex task requiring the interpretation of spoken language exclusively from visual cues, primarily lip movements and facial expressions. This task is especially challenging due to the absence of auditory information and the inherent ambiguity when visually distinguishing phonemes that have overlapping visemes where different phonemes appear identical on the lips. Current methods typically attempt to predict words or characters directly from these visual cues, but this approach frequently encounters high error rates due to coarticulation effects and viseme ambiguity. We propose a novel two-stage, phoneme-centric framework for Visual Automatic Speech Recognition (V-ASR) that addresses these longstanding challenges. First, our model predicts a compact sequence of phonemes from visual inputs using a Video Transformer with a CTC head, thereby reducing the task complexity and achieving robust speaker invariance. This phoneme output then serves as the input to a fine-tuned Large Language Model (LLM), which reconstructs coherent words and sentences by leveraging broader linguistic context. Unlike existing methods that either predict words directly-often faltering on visually similar phonemes-or rely on large-scale multimodal pre-training, our approach explicitly encodes intermediate linguistic structure while remaining highly data efficient. We demonstrate state-of-the-art performance on two challenging datasets, LRS2 and LRS3, where our method achieves significant reductions in Word Error Rate (WER) achieving a SOTA WER of 18.7 on LRS3 despite using 99.4% less labelled data than the next best approach.

Introduction

Lip reading (V-ASR) is notoriously difficult due to "visemes"—phonemes that look identical on the lips (e.g., the sounds 'p', 'b', and 'm')—and "coarticulation," where the visual appearance of a sound changes based on its surrounding context. Traditional end-to-end models attempt to map video frames directly to text, requiring thousands of hours of labeled data to learn visual patterns and language context simultaneously. VALLR addresses this by decoupling the visual task from the linguistic task through a phoneme-centric approach.

Methods

VALLR utilizes a modular two-stage architecture designed for data efficiency and interpretability.

VALLR Architecture: The two-stage framework consists of a Video-to-Phoneme stage using a ViT and CTC head, followed by a Phoneme-to-Sentence stage using a fine-tuned LLM.

Stage 1: Video-to-Phoneme

The visual front-end employs a Vision Transformer (ViT-base) to process cropped video of the speaker's mouth. An adapter consisting of 1D convolutions downsamples the temporal dimension, followed by a Connectionist Temporal Classification (CTC) head that predicts a sequence of 38 English phonemes. This stage focuses purely on visual-to-phonetic mapping, which is largely speaker-invariant.

Stage 2: Phoneme-to-Sentence

The resulting noisy phoneme sequence is fed into a fine-tuned Large Language Model (specifically Llama 3.2-3B). Using Low-Rank Adaptation (LoRA), the LLM is trained to "translate" phoneme sequences into coherent English sentences. This allows the model to leverage its internal linguistic knowledge to resolve visual ambiguities, such as distinguishing between 'p' and 'b' based on global sentence context.

Results

The model was evaluated on the LRS2 (BBC footage, ~224 hours) and LRS3 (TED talks, ~439 hours) datasets. To enhance the LLM's reconstruction capabilities, the authors also utilized a synthetic corpus generated from WikiText to pre-train the phoneme-to-sentence mapping without requiring additional video data.

Performance Comparison: VALLR (Ours) achieves the lowest WER while requiring significantly fewer labeled training hours compared to existing self-supervised and fully supervised methods.

VALLR achieved a state-of-the-art 18.7% WER on the LRS3 dataset. Remarkably, it reached this performance using only 30 hours of labeled video data—a 99.4% reduction compared to previous top-performing models like ViT-3D, which required 90,000 hours. Ablation studies confirmed that the phoneme-centric bottleneck is critical; removing it and training an end-to-end ViT+LLM model increased the WER to 25.6%. Furthermore, scaling the language model from GPT-2 Small to Llama 3.2-3B yielded a 15% improvement in WER.

Conclusion

VALLR demonstrates that a "phoneme bottleneck" allows for extreme data efficiency in lip reading by leveraging the pre-existing linguistic knowledge of LLMs. While the model still faces challenges with homophones and subtle visual distinctions (e.g., /s/ vs /z/), it sets a new benchmark for V-ASR. Future work will explore speaker-adaptive fine-tuning and improved multi-frame alignment to further refine visual recognition.

Summary updated

Abstract

Lip Reading, or Visual Automatic Speech Recognition (V-ASR), is a complex task requiring the interpretation of spoken language exclusively from visual cues, primarily lip movements and facial expressions. This task is especially challenging due to the absence of auditory information and the inherent ambiguity when visually distinguishing phonemes that have overlapping visemes where different phonemes appear identical on the lips. Current methods typically attempt to predict words or characters directly from these visual cues, but this approach frequently encounters high error rates due to coarticulation effects and viseme ambiguity. We propose a novel two-stage, phoneme-centric framework for Visual Automatic Speech Recognition (V-ASR) that addresses these longstanding challenges. First, our model predicts a compact sequence of phonemes from visual inputs using a Video Transformer with a CTC head, thereby reducing the task complexity and achieving robust speaker invariance. This phoneme output then serves as the input to a fine-tuned Large Language Model (LLM), which reconstructs coherent words and sentences by leveraging broader linguistic context. Unlike existing methods that either predict words directly-often faltering on visually similar phonemes-or rely on large-scale multimodal pre-training, our approach explicitly encodes intermediate linguistic structure while remaining highly data efficient. We demonstrate state-of-the-art performance on two challenging datasets, LRS2 and LRS3, where our method achieves significant reductions in Word Error Rate (WER) achieving a SOTA WER of 18.7 on LRS3 despite using 99.4% less labelled data than the next best approach.
BibTeX
@inproceedings{thomas2025vallr,
  author = {Marshall Thomas and Edward Fish and Richard Bowden},
  title = {{VALLR: Visual ASR Language Model for Lip Reading}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure