Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

Qiushi Zhu, Jie Zhang, Yu Gu, Yuchen Hu, Lirong DaiAAAI 2024 · DOI

Introduction

Self-supervised learning (SSL) has significantly advanced speech processing, particularly in near-field scenarios. However, far-field environments—characterized by high ambient noise and reverberation—remain a substantial challenge. While visual information (lip movements) and multichannel audio (spatial filtering) are known to improve robustness, existing SSL frameworks often fail to effectively integrate these modalities for far-field applications. This paper introduces AV-wav2vec2, a novel framework designed to learn robust speech representations by leveraging both visual cues and the spatial redundancy of multichannel audio.

Proposed Framework

AV-wav2vec2 extends the wav2vec 2.0 architecture into a multi-path, multi-modal system. It processes a visual stream alongside multiple audio streams in parallel.

AV-wav2vec2 pre-training architecture featuring inter-channel contrastive loss.

Inter-Channel Contrastive Loss

The primary innovation is the inter-channel contrastive loss ($L_{c2}$). Unlike standard SSL, which predicts masked segments from local context, AV-wav2vec2 attempts to predict the masked segment of one channel using features from other channels. Since speech content is consistent across microphones while noise varies spatially, this objective forces the model to extract noise-invariant speech features.

Joint Training and Robustness

To address the scarcity of labeled multichannel data, the model is jointly trained on large-scale single-channel datasets like WenetSpeech. Furthermore, modality dropout is employed during training—randomly omitting video or specific audio channels—to ensure the model remains robust to sensor failures in real-world deployments.

Experimental Setup

The framework was pre-trained and evaluated using the following datasets:

  • MISP2021-AVSR: A Chinese dataset with 141 hours of audio-visual data recorded in real home environments, including far-field (6-channel), middle-field (2-channel), and near-field audio.
  • WenetSpeech: 900 hours of unlabeled Mandarin YouTube audio used for the joint training phase.

Results and Discussion

AV-wav2vec2 was evaluated across several downstream tasks, demonstrating superior performance in challenging environments.

Audio-Visual Speech Recognition (AVSR)

On the most difficult far-field test set (C4), AV-wav2vec2 achieved a Character Error Rate (CER) of 69.47%. This represents a significant improvement over the MS-TCN baseline (84.21%) and a modified AV-HuBERT (89.84%). The results suggest that visual information is particularly beneficial in low Signal-to-Noise Ratio (SNR) scenarios.

Audio-Visual Speaker Diarization (AVSD)

As a feature extractor, the model achieved a Diarization Error Rate (DER) of 10.44%, outperforming both supervised baselines (12.97%) and other state-of-the-art SSL models like VATLM (10.85%).

Ablation Study

The importance of the inter-channel contrastive loss was confirmed through ablation testing across different environmental configurations.

Impact of inter-channel contrastive loss on CER across far-field test conditions.

Removing $L_{c2}$ led to a performance degradation, with CER increasing from 70.65% to 73.12% on the C4 test set. This confirms that cross-microphone learning is essential for filtering out environmental noise and improving far-field speech recognition accuracy.

Conclusion

AV-wav2vec2 demonstrates that combining visual information with the spatial redundancy of multichannel audio through inter-channel contrastive learning creates highly robust speech representations. While challenges like domain mismatch and polyphony in Chinese speech persist, this framework provides a strong foundation for future far-field multi-modal speech processing research.

Summary updated

BibTeX
@inproceedings{zhu2024multichannel,
  author = {Qiushi Zhu and Jie Zhang and Yu Gu and Yuchen Hu and Lirong Dai},
  title = {{Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29951},
}
Figure