Abstract
In this paper, we propose a novel method for speaker adaptation in lip reading, motivated by two observations. Firstly, a speaker's own characteristics can always be portrayed well by his/her few facial images or even a single image with shallow networks, while the fine-grained dynamic features associated with speech content expressed by the talking face always need deep sequential networks to represent accurately. Therefore, we treat the shallow and deep layers differently for speaker adaptive lip reading. Secondly, we observe that a speaker's unique characteristics (e.g. prominent oral cavity and mandible) have varied effects on lip reading performance for different words and pronunciations, necessitating adaptive enhancement or suppression of the features for robust lip reading. Based on these two observations, we propose to take advantage of the speaker's own characteristics to automatically learn separable hidden unit contributions with different targets for shallow layers and deep layers respectively. Our approach consistently outperforms existing methods, as confirmed by comprehensive analysis and comparison across different settings. Besides the evaluation on the popular LRW-ID and GRID datasets, we also release a new dataset for evaluation, CAS-VSR-S68h, to further assess the performance in an extreme setting where just a few speakers are available but the speech content covers a large and diversified range.
Introduction
Lip-reading, or Visual Speech Recognition (VSR), is notoriously difficult because of the high variability between speakers. Factors like mouth shape, facial structure, and even personal "speaking styles" act as a form of noise that obscures the actual linguistic content. Current models often struggle with "unseen" speakers—people not included in the training data. While general adaptation techniques exist, they often treat the entire neural network as a black box. This paper addresses the need for a more nuanced approach that recognizes how speaker identity and speech content are represented at different stages of a deep learning model.
Methodology
The authors propose a "Separable Hidden Unit Contribution" approach, inspired by the LHUC method used in audio speech recognition. Their architecture consists of four main modules:
- Speaker Verification Module: A dedicated sub-network (using (2+1)D Convolutions and GRUs) that extracts a unique "speaker feature" ($\theta$) from the video.
- Lip Reading Module: The backbone network (ResNet-18 + GRU) that performs the actual task of recognizing words.
- Feature Enhancement Module (Shallow Layers): In the early layers of the network, speech content features are often weak compared to dominant facial traits. This module uses the speaker feature to "boost" the relevant content signals.
- Feature Suppression Module (Deep Layers): In deeper layers, where content is well-represented, the speaker's unique traits can become a distraction. This module uses the speaker feature to "mute" these irrelevant signals.
The system is optimized using a combination of Triplet Loss to ensure speaker features are distinct and standard VSR losses like Cross-Entropy or CTC.
Experimental Setup
The researchers evaluated their method on three key datasets:
- GRID: A standard, small-scale dataset with 33 speakers.
- LRW-ID: A re-annotated version of the "Lip Reading in the Wild" dataset, specifically organized to test speaker adaptation across thousands of individuals.
- CAS-VSR-S68 (New): A newly released benchmark featuring 68 hours of high-resolution news broadcast footage. It is designed for "extreme" settings: it has very few speakers but a massive, diversified vocabulary of over 3,800 Chinese characters.
Results and Discussion
The proposed method consistently outperformed existing state-of-the-art (SOTA) techniques like User-padding and Prompt Tuning.
The Depth-Identity Trade-off
A key insight of this work is the observation that shallow networks are proficient at identifying who is talking, but struggle with what is being said. Conversely, deep networks are needed to capture the fine-grained temporal dynamics of speech. This motivated the layer-specific treatment of features.
Key Performance Metrics
- LRW-ID: Achieved 90.45% accuracy with just 5 minutes of adaptation data.
- GRID: Reduced the Word Error Rate (WER) to a remarkable 3.59% with 5 minutes of adaptation.
- Zero-Shot Performance: Even without specific adaptation data for a new speaker, the model outperformed the baseline, showing better generalization "out of the box."
Qualitative Analysis
The model learns to treat speaker traits as "dynamic filters." A speaker's physical attributes (like outward-turning lips) can dampen or amplify specific sounds; the model learns to compensate for these physical "biases" adaptively. Qualitative heatmaps revealed that the enhancement module often focuses on the chin, cheeks, and nose—areas containing significant auxiliary information about jaw movement and facial tension during speech.
Conclusion
This research demonstrates that treating neural network layers differently based on their depth—enhancing content in shallow layers and suppressing noise in deep layers—significantly improves speaker-adaptive lip-reading. While highly effective, the model requires a minimum threshold of "calibration" time in very high-diversity linguistic environments. Future work may involve exploring how these separable contributions could be applied to even larger transformer-based architectures or multi-modal systems.
Summary updated