Speaker Adaptation for Lip-Reading Using Visual Identity Vectors

Pujitha Appan Kandala, Abhinav Thanda, Dilip Kumar Margam, Rohith Chandrashekar Aralikatti, Tanay Sharma, Sharad Roy, Shankar M. VenkatesanInterspeech 2019

TL;DR

This paper introduces a novel method for improving lip-reading accuracy on unseen speakers by adapting existing models using "visual i-vectors." These vectors, derived from facial features, capture speaker-specific lip movement characteristics, significantly reducing word error rates and making lip-reading systems more robust to individual variations.

What Problem Does This Paper Address?

Lip-reading, or visual speech recognition, is inherently challenging due to the limited visual information available from mouth movements. While deep learning has improved performance, existing models still struggle with high word error rates (WER), especially when encountering speakers they haven't been trained on. This "unseen speaker" problem is a significant hurdle for deploying lip-reading systems in real-world scenarios. The paper aims to bridge this gap by adapting lip-reading models to individual speakers, similar to how speaker adaptation is used in traditional Automatic Speech Recognition (ASR).

Methods

The core innovation lies in the use of visual i-vectors for speaker adaptation. These vectors are analogous to i-vectors used in ASR, which capture speaker characteristics. Here, they are extracted from visual features of lip movements. The process involves:

  • Feature Extraction: 2D-DCT features are extracted from lip images.
  • UBM (Universal Background Model) Training: A Gaussian Mixture Model (GMM) or an RNN-HMM based model is trained on the features from all speakers to create a general representation.
  • Visual i-vector Estimation: Factor analysis is applied to the speaker's features and the UBM to derive a low-dimensional "visual i-vector" that encapsulates their unique lip movement patterns. Two methods for estimating sufficient statistics for this process are explored: a standard GMM-UBM and an ancillary UBM derived from an RNN-HMM model.
  • Model Adaptation: The extracted visual i-vector is then fed as an additional input to the hidden layers of the lip-reading model (a 3D-CNN, 2D-CNN, BiLSTM architecture trained with Connectionist Temporal Classification - CTC loss) during both training and testing. This allows the model to learn speaker-specific nuances.

The paper contrasts this with traditional model or feature adaptation methods, highlighting that visual i-vectors can be extracted offline and integrated seamlessly.

Datasets

  • GRID Audio-Visual Corpus: This is the primary dataset used for training and evaluation. It contains audio-visual data from 34 speakers. The authors note that data for speaker 21 was unavailable.
  • Korean Visual Corpus: A custom-created corpus of Korean language sentences spoken by 66 speakers was also used to validate the approach on a different language.

Key Insights and Ideas

  • Visual i-vectors as Speaker Signatures: The paper demonstrates that, much like in ASR, a compact vector can effectively capture the unique characteristics of a speaker's lip movements.
  • Seamless Integration: Incorporating visual i-vectors as an additional input to the lip-reading network is a straightforward yet powerful adaptation technique.
  • RNN-HMM Superiority for Statistics: The study suggests that using statistics derived from an RNN-HMM model for i-vector estimation yields slightly better results than a standard GMM-UBM.
  • Dimensionality Matters: The performance improvement from visual i-vectors is sensitive to their dimensionality, with an optimal dimension of 10 found to be most effective.

Key Results

  • Significant WER Reduction: The proposed visual i-vector adaptation method achieved substantial improvements on the GRID corpus for unseen speakers.
  • Using UBM-based visual i-vectors resulted in a 10.4% relative improvement in WER over the baseline.
  • Visual i-vectors derived from UBMTS/UBMPH models provided even greater improvements of 11.6% and 15% respectively, compared to the baseline.
  • Cross-Lingual Validation: Similar performance improvements were observed on the Korean corpus, with gains of 5.6% and 8.7% using UBMTS/UBMPH derived i-vectors.
  • Data Requirements: For optimal performance on the GRID corpus, approximately 25 minutes of unseen speaker data was found to be necessary for visual i-vector estimation.

Limitations and Future Work

  • Limitations:
    • The study assumes a constant frontal pose for speakers, and variations due to facial hair or makeup were not addressed.
  • The optimal dimensionality of the visual i-vector was empirically determined and might vary for different datasets or architectures.
  • Future Work:
    • The authors plan to explore separating lip shape and texture for more refined i-vector extraction.
  • They aim to address variations in facial features and different speaker poses.
  • A natural extension is to apply this factor analysis-based adaptation method to audio-visual speech recognition systems.

Summary updated

BibTeX
@inproceedings{kandala2019speaker,
  author = {Pujitha Appan Kandala and Abhinav Thanda and Dilip Kumar Margam and Rohith Chandrashekar Aralikatti and Tanay Sharma and Sharad Roy and Shankar M. Venkatesan},
  title = {{Speaker Adaptation for Lip-Reading Using Visual Identity Vectors}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure