Space-time audio-visual speech recognition with multiple multi-class probabilistic support vector machines

Samuel Pachoud, Shaogang Gong, Andrea CavallaroAVSP 2009

TL;DR

This paper presents a novel audio-visual speech recognition system that leverages multiple probabilistic Support Vector Machines (SVMs) to effectively fuse audio and visual cues, even in noisy conditions. By dynamically adapting the fusion strategy based on confidence factors from individual modalities, the system achieves robust digit recognition, outperforming existing methods.

What Problem Does This Paper Address?

Recognizing speech from audio and visual signals (lip movements) simultaneously, known as Audio-Visual Speech Recognition (AVSR), is a challenging task, especially in real-world scenarios where signals are often degraded by noise. Traditional AVSR systems often struggle with integrating information from these two modalities, particularly when one or both are compromised. The core problem this paper tackles is how to effectively combine audio and visual speech information in a robust manner, allowing the system to rely more on the cleaner modality when the other is degraded, and to achieve accurate recognition even when both are noisy. This requires not only extracting relevant features but also intelligently fusing them based on their reliability.

Methods

The authors propose a system that utilizes multiple multi-class probabilistic Support Vector Machines (SVMs). Here's a breakdown of their approach:

  • Feature Extraction:
    • Visual: Spatio-temporal features are extracted from lip movements using 2D + time SIFT descriptors.
  • Audio: Mel-Frequency Cepstral Coefficients (MFCCs) and Spectral Flatness Measure (SFM) are used. Dimensionality reduction is performed using Principal Component Analysis (PCA).
  • Confidence Factor Generation: Two single-modality classifiers (one for audio, one for visual) are trained. These classifiers are probabilistic SVMs, meaning their outputs are not just class predictions but also provide a measure of confidence. These confidence factors are crucial for determining the reliability of each modality.
  • Adaptive Fusion Strategy: Based on the confidence factors, the system dynamically selects one of three fusion strategies:
  • If visual is noisy and audio is clean, it prioritizes audio.
  • If audio is noisy and visual is clean, it prioritizes visual.
  • If both are degraded or clean, it uses a combined approach. The fusion itself is performed using kernel Canonical Correlation Analysis (kCCA), which finds canonical factors to represent the joint audio-visual space.

  • Recognition: A final multi-class SVM is trained on the fused audio-visual feature space to perform the digit recognition. The paper uses a "one-versus-the-rest" strategy for their multi-class SVMs, which they found to be empirically effective.

Datasets

The evaluation was performed on the CUAVE database. This database consists of 36 speakers pronouncing 10 connected digits. It includes over 2500 utterances with both audio and visual speech signals. The authors also introduced specific degradation conditions to the database for evaluation:

  • Visual Degradation: Partial occlusion (fingers covering parts of the mouth) and salt-and-pepper noise.
  • Audio Degradation: Additive Gaussian noise at various Signal-to-Noise Ratios (SNRs).

Key Insights and Ideas

  • Probabilistic SVMs for Confidence: The core innovation lies in using probabilistic SVMs not just for classification but to derive confidence scores for each modality. This allows the system to intelligently decide which modality to trust more.
  • Adaptive Fusion is Key: Instead of a fixed fusion strategy, dynamically adapting the fusion based on signal quality is crucial for robustness in degraded conditions.
  • Visual Features Cope Well with Occlusion: The spatio-temporal visual features (SIFT descriptors) proved to be relatively robust to partial visual occlusion, highlighting the power of well-chosen visual features.
  • Confidence Factors Mitigate Noise Dependency: The confidence factors effectively reduce the system's dependency on the quality of the audio signal, leading to more stable performance across different noise levels.

Key Results

The paper demonstrates that their proposed system achieves high recognition rates even under significant audio and visual degradation.

  • For example, with visual occlusion of 19% and an audio SNR of -5dB, the recognition rate was still around 28.3%.
  • In less degraded conditions (e.g., clean video, SNR of 25dB), the system achieved recognition rates of up to 97.3%.
  • The authors highlight that visual occlusion was less disturbing than salt-and-pepper noise, and that their confidence factor mechanism significantly improved performance in noisy audio conditions compared to systems that don't account for signal quality.

Limitations and Future Work

  • Limitations: The paper acknowledges that a complete comparative evaluation against all existing methods is difficult due to the lack of studies on degraded audio-visual signals. They also note that their current approach focuses on digit recognition, and extending it to more complex speech tasks would be a natural next step.
  • Future Work: The authors suggest exploring other types of audio degradation, such as compression and reverberation, as future research directions. They also plan to investigate more complex speech recognition tasks beyond digit recognition.

Summary updated

BibTeX
@inproceedings{pachoud2009space,
  author = {Samuel Pachoud and Shaogang Gong and Andrea Cavallaro},
  title = {{Space-time audio-visual speech recognition with multiple multi-class probabilistic support vector machines}},
  booktitle = {AVSP},
  year = {2009},
}
Figure