Feature selection for pose invariant lip biometrics

Adrian Pass, Jianguo Zhang, Darryl StewartInterspeech 2010 · OpenReview

Abstract

For the first time in this paper we present results showing the effect of out of plane speaker head pose variation on a lip based speaker verification system. Using appearance DCT based features, we adopt a Mutual Information analysis technique to highlight the class discriminant DCT components most robust to changes in out of plane pose. Experiments are conducted using the initial phase of a new multi view Audio-Visual database designed for research and development of pose-invariant speech and speaker recognition. We show that verification performance can be improved by substituting higher order horizontal DCT components for vertical, particularly in the case of a train/test pose angle mismatch. We further show that the best performance can be achieved by combining this alternative feature selection with multi view training, reporting a relative 45% Equal Error Rate reduction over a common energy based selection.

Introduction

Lip biometrics—using the movement and shape of the mouth to verify identity—is a powerful tool, especially when combined with audio in noisy environments. However, most systems assume a "cooperative" user looking straight into a camera. In real-world scenarios, such as doorway access control, a user’s head might be tilted or turned (out-of-plane pose variation). This change in perspective "squashes" the horizontal appearance of the mouth, causing standard feature extraction methods to fail. This paper aims to identify a set of features that remain invariant or highly robust to these changes in viewing angle.

Methods

The study focuses on Discrete Cosine Transform (DCT) features, standard for compact mouth appearance representation.

Mutual Information (MI) Analysis

Instead of standard energy-based "triangular" selection, the authors use MI to identify DCT coefficients containing the most speaker identity information across different angles.

Feature Masking

Three selection masks were compared:

  • 'tri' (Baseline): Standard energy-based triangular selection.
  • 'even': Retains only even columns to force horizontal symmetry.
  • '3col' (Proposed): Based on MI analysis, prioritizing vertical frequency components by focusing on the first three even columns.
DCT coefficient selection masks for frontal and multi-angle (0-30 degrees) data.

Modeling

Gaussian Mixture Models (GMMs) were used for speaker modeling, with a Universal Background Model (UBM) for likelihood ratio normalization.

Key Insights

When a speaker turns their head horizontally, the mouth image undergoes significant compression in the horizontal plane. However, vertical movements remain relatively consistent. By intentionally discarding higher-order horizontal DCT components (the "3col" mask), the system effectively "smears" horizontal detail. This acts as a form of normalization, making the 30-degree view look more like the 0-degree view to the classifier.

Visual comparison of original and reconstructed lip images across different poses and selection methods.

Results

The proposed '3col' feature selection achieved a 45% relative reduction in Equal Error Rate (EER) compared to the 'tri' selection across multiple views. Multi-pose training (0–30 degrees) also significantly improved performance, dropping frontal-view EER from 12.2% to 3.3% using the 'tri' mask. The best overall performance (1.67% EER) was achieved by combining the '3col' mask with multi-view training.

Conclusion

While the QuLips dataset used is small (3 subjects), the results demonstrate the potential of MI-based feature selection for pose invariance. Future work will involve larger datasets and investigating vertical rotation (pitch) and complex movements.

Summary updated

Abstract

For the first time in this paper we present results showing the effect of out of plane speaker head pose variation on a lip based speaker verification system. Using appearance DCT based features, we adopt a Mutual Information analysis technique to highlight the class discriminant DCT components most robust to changes in out of plane pose. Experiments are conducted using the initial phase of a new multi view Audio-Visual database designed for research and development of pose-invariant speech and speaker recognition. We show that verification performance can be improved by substituting higher order horizontal DCT components for vertical, particularly in the case of a train/test pose angle mismatch. We further show that the best performance can be achieved by combining this alternative feature selection with multi view training, reporting a relative 45% Equal Error Rate reduction over a common energy based selection.
BibTeX
@inproceedings{pass2010feature,
  author = {Adrian Pass and Jianguo Zhang and Darryl Stewart},
  title = {{Feature selection for pose invariant lip biometrics}},
  booktitle = {Interspeech},
  year = {2010},
  url = {https://openreview.net/forum?id=CkiJ79c2Hi},
}
Figure