Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition

Zi-qiang Zhang, Jie Zhang, Jianshu Zhang, Ming-Hui Wu, Xin Fang, Li-Rong DaiICIP 2022 · OpenReview

TL;DR

This paper introduces a novel self-supervised learning framework for audio-visual speech recognition (AVSR) that effectively fuses information from both audio and visual modalities while capturing long-term temporal dependencies. The proposed method leverages a transformer-based fusion module and a flexible masking strategy to learn contextually rich representations, which can then be applied to AVSR and even single-modal tasks like ASR and lipreading with minimal adaptation.

What Problem Does This Paper Address?

Audio-visual speech recognition (AVSR) holds great promise for improving speech understanding, especially in noisy environments, by combining auditory and visual cues. However, training effective AVSR models typically requires large amounts of manually labeled data, which is expensive and time-consuming to obtain. While self-supervised learning (SSL) has shown success in learning representations from unlabeled data for individual modalities (like audio or visual), effectively fusing these modalities in an SSL setting for AVSR remains a challenge. Existing methods often struggle to capture long-term context dependencies crucial for speech recognition and may treat modalities too separately, limiting cross-modal information flow. This paper aims to bridge this gap by developing an SSL approach that learns fused audio-visual representations capable of capturing both inter-modal complementarity and temporal context.

Methods

The core of the proposed method is a self-supervised learning framework that extends the principles of Wav2vec 2.0 to the audio-visual domain. The framework consists of three main components:

  • Audio and Visual Encoders: Standard encoders (e.g., 1D CNNs for audio, 3D CNNs + ResNet for visual) process the raw audio and visual inputs, respectively, downsampling them to a synchronized frame rate.
  • Transformer-based Fusion Module: This module takes the encoded audio and visual embeddings and learns to fuse them, capturing both cross-modal interactions and temporal context. It uses a multi-layer perceptron (MLP) followed by transformer blocks. Crucially, it concatenates the audio and visual embeddings in the feature dimension before feeding them into the transformer blocks to preserve frame-level alignment.
  • Flexible Masking Strategy and Audio-Visual Recovery Objective: The model is trained to "recover" corrupted audio and visual embeddings. A flexible masking strategy is employed, where parts of the audio and/or visual embeddings are masked out. The model then uses the remaining information (from the unmasked modality and temporal context) to reconstruct the masked embeddings. This recovery is guided by a contrastive loss, where the target embeddings are treated as positive samples. The paper highlights the importance of including "cross-modal negative samples" in the contrastive loss, which helps to pull the audio, visual, and fused embeddings into a unified representation space.

A key innovation is the "audio-visual recovery" objective, which unifies audio-visual correspondence and masked prediction. The flexibility of the masking strategy allows the model to learn from scenarios where one modality is available and the other is not, making it robust for downstream tasks.

Datasets

  • Pre-training: The model is pre-trained on the LRS3 [29] dataset, specifically the "pretrain" subset, which contains approximately 408 hours of unlabeled video speech.
  • Downstream Tasks:
    • AVSR: Fine-tuned on the "trainval" subset of LRS3 [29], which has about 30 hours of labeled data.
  • ASR & VSR: Fine-tuned on the same 30 hours of labeled LRS3 data.
  • Lipreading: Evaluated on the LRW [28] dataset, which contains about 173 hours of data for 500 spoken words.

Key Insights and Ideas

  • Audio Dominance in Self-Supervised Learning: The paper observes that audio signals tend to be more informative than visual signals for learning speech representations in a self-supervised manner. This is evidenced by better performance when pre-training with audio-only compared to visual-only.
  • Complementarity of Modalities Aids Single-Modal Tasks: Even when a downstream task only uses one modality (e.g., ASR), pre-training with both audio and visual data leads to improved performance. This suggests that the auxiliary modality helps learn more robust and generalizable representations.
  • Masked Embedding as a Modality Replacement: The learned "mask embedding" can effectively substitute for a missing modality during fine-tuning for single-modal tasks, demonstrating the flexibility of the learned representations.
  • Importance of Cross-Modal Negatives: Including cross-modal negative samples in the contrastive loss is crucial for pushing audio, visual, and fused embeddings into a unified space, leading to better overall representation learning.
  • Fusion Module's Role in Context and Complementarity: The transformer-based fusion module is key to capturing both the complementary information between modalities and the long-term temporal context within the sequences.

Key Results

  • Improved AVSR Performance: The proposed method significantly improves AVSR performance, reducing Word Error Rate (WER) by up to 13.6% relative to single-modal baselines.
  • State-of-the-Art Lipreading: The pre-trained model, when fine-tuned for lipreading, achieves state-of-the-art performance among self-supervised methods and outperforms its supervised counterpart by 3.1% accuracy.
  • Robustness to Modality Exclusion: The pre-trained model shows strong performance on single-modal tasks (ASR, VSR) even when one modality is excluded during pre-training, and replacing a modality with a learned mask embedding proves effective.
  • Ablation Studies: Experiments show that jointly optimizing losses for both modalities (Lca + Lcv) improves performance, and the addition of cross-modal negative samples further boosts results. Extending pre-training iterations (D3) also leads to performance gains.

Limitations and Future Work

  • Audio Dominance Imbalance: The authors acknowledge that the audio modality tends to dominate in audio-visual self-supervised learning, potentially leading to an imbalance between the learned representations of the two modalities.
  • Future Work: They suggest that this imbalance could be addressed by training with larger datasets or employing stronger data augmentation techniques for audio/visual inputs. Further research could explore more advanced fusion strategies and masking techniques.

Summary updated

Abstract

With the advance in self-supervised learning for audio and visual modalities, it has become possible to learn a robust audio-visual speech representation. This would be beneficial for improving the audio-visual speech recognition (AVSR) performance, as the multi-modal inputs contain more fruitful information in principle. In this paper, based on existing self-supervised representation learning methods for audio modality, we therefore propose an audio-visual representation learning approach. The proposed approach explores both the complementarity of audio-visual modalities and long-term context dependency using a transformer-based fusion module and a flexible masking strategy. After pre-training, the model is able to extract fused representations required by AVSR. Without loss of generality, it can be applied to single-modal tasks, e.g. audio/visual speech recognition by simply masking out one modality in the fusion module. The proposed pre-trained model is evaluated on speech recognition and lipreading tasks using one or two modalities, where the superiority is revealed.
BibTeX
@inproceedings{zhang2022learning,
  author = {Zi-qiang Zhang and Jie Zhang and Jianshu Zhang and Ming-Hui Wu and Xin Fang and Li-Rong Dai},
  title = {{Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition}},
  booktitle = {ICIP},
  year = {2022},
  url = {https://openreview.net/forum?id=4PMnq38YU0},
}
Figure