Abstract
For the first time in this paper we present results showing the effect of out of plane speaker head pose variation on a lip based speaker verification system. Using appearance DCT based features, we adopt a Mutual Information analysis technique to highlight the class discriminant DCT components most robust to changes in out of plane pose. Experiments are conducted using the initial phase of a new multi view Audio-Visual database designed for research and development of pose-invariant speech and speaker recognition. We show that verification performance can be improved by substituting higher order horizontal DCT components for vertical, particularly in the case of a train/test pose angle mismatch. We further show that the best performance can be achieved by combining this alternative feature selection with multi view training, reporting a relative 45% Equal Error Rate reduction over a common energy based selection.
Introduction
Lip biometrics—using the movement and shape of the mouth to verify identity—is a powerful tool, especially when combined with audio in noisy environments. However, most systems assume a "cooperative" user looking straight into a camera. In real-world scenarios, such as doorway access control, a user’s head might be tilted or turned (out-of-plane pose variation). This change in perspective "squashes" the horizontal appearance of the mouth, causing standard feature extraction methods to fail. This paper aims to identify a set of features that remain invariant or highly robust to these changes in viewing angle.
Methods
The study focuses on Discrete Cosine Transform (DCT) features, standard for compact mouth appearance representation.
Mutual Information (MI) Analysis
Instead of standard energy-based "triangular" selection, the authors use MI to identify DCT coefficients containing the most speaker identity information across different angles.
Feature Masking
Three selection masks were compared:
- 'tri' (Baseline): Standard energy-based triangular selection.
- 'even': Retains only even columns to force horizontal symmetry.
- '3col' (Proposed): Based on MI analysis, prioritizing vertical frequency components by focusing on the first three even columns.
Modeling
Gaussian Mixture Models (GMMs) were used for speaker modeling, with a Universal Background Model (UBM) for likelihood ratio normalization.
Key Insights
When a speaker turns their head horizontally, the mouth image undergoes significant compression in the horizontal plane. However, vertical movements remain relatively consistent. By intentionally discarding higher-order horizontal DCT components (the "3col" mask), the system effectively "smears" horizontal detail. This acts as a form of normalization, making the 30-degree view look more like the 0-degree view to the classifier.
Results
The proposed '3col' feature selection achieved a 45% relative reduction in Equal Error Rate (EER) compared to the 'tri' selection across multiple views. Multi-pose training (0–30 degrees) also significantly improved performance, dropping frontal-view EER from 12.2% to 3.3% using the 'tri' mask. The best overall performance (1.67% EER) was achieved by combining the '3col' mask with multi-view training.
Conclusion
While the QuLips dataset used is small (3 subjects), the results demonstrate the potential of MI-based feature selection for pose invariance. Future work will involve larger datasets and investigating vertical rotation (pitch) and complex movements.
Summary updated