Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection In Low-Resource Realistic Scenarios

Ya Jiang, Qing Wang, Jun Du, Maocheng Hu, Pengfei Hu, Zeyan Liu, Shi Cheng, Zhaoxu Nian, Yuxuan Dong, Mingqi Cai, Xin Fang, Chin-Hui LeeICME 2024 · OpenReview

TL;DR

This paper tackles the challenging problem of Sound Event Localization and Detection (SELD) in realistic, low-resource audio-visual scenarios. They introduce a novel framework combining cross-modal teacher-student learning, a two-stage audio-visual fusion strategy, and a new data augmentation technique (VPS) to significantly improve performance, ultimately winning first place in the DCASE 2023 Challenge.

What Problem Does This Paper Address?

Sound Event Localization and Detection (SELD) aims to pinpoint when and where specific sounds occur in an audio scene. This is crucial for applications like intelligent surveillance, smart homes, and assistive technologies. While audio-only SELD has seen progress, integrating visual information (audio-visual SELD or AV-SELD) holds great promise for richer understanding. However, real-world AV-SELD faces significant hurdles: limited availability of synchronized audio-visual data, privacy concerns blurring faces in videos, and the inherent challenge of mapping 3D audio cues to 2D video. This paper specifically addresses the "low-resource" aspect, where training data is scarce, making it difficult to effectively leverage the visual modality.

Methods

The authors propose a multi-pronged approach:

  • Cross-Modal Teacher-Student Learning (TSL): To overcome data scarcity, they train a powerful "teacher" model on abundant, purely audio data. This teacher model then guides a "student" model that's trained on limited audio-visual data. This knowledge transfer helps the student model learn robust audio features and generalize better.
  • Two-Stage Audio-Visual Fusion:
    • Early Feature Fusion: Audio and visual features are combined early in the network to capture synergistic information.
  • Late Video-Guided Decision Fusion (VGDF): This is a novel post-processing step. It uses detected human keypoints (like mouths) from the video to refine the sound source localization (DOA) estimates from the audio-visual model. If visual cues strongly suggest a sound's origin, they override the audio-based prediction, particularly for events like speech.
  • Audio-Visual Joint Data Augmentation (ACS-VPS): To further boost robustness and effectively use the limited data, they introduce Video Pixel Swapping (VPS). This technique extends the existing Audio Channel Swapping (ACS) method to the audio-visual domain. VPS creatively manipulates video frames (e.g., rotating, flipping) to create diverse training samples, effectively multiplying the training data.

Datasets

  • DCASE 2023 Challenge development set (STARSS23): This is the primary dataset used for training and evaluation. It's noted as being a "low-resource" dataset with only 3.83 hours of training data.
  • Additional simulated audio data: Used to train the initial, data-rich teacher model.

Key Insights and Ideas

  • Teacher-student learning is powerful for low-resource AV-SELD: Transferring knowledge from a well-trained audio model to an AV student model is a highly effective strategy when AV data is scarce.
  • Visual cues can significantly refine audio localization: The video-guided decision fusion (VGDF) demonstrates that leveraging visual information, even in a post-processing step, can correct audio-based localization errors and improve overall performance.
  • Creative data augmentation is crucial: The proposed ACS-VPS method is key to making the most of limited audio-visual data, creating diverse scenarios that improve model robustness.
  • Audio-visual fusion offers a tangible benefit: Even with limited data, AV models outperform AO models, highlighting the inherent value of multi-modal information.

Key Results

  • The proposed AV-SELD framework achieved a SELD score of 0.28, outperforming the second-place submission in the DCASE 2023 Challenge by a significant 15.2%.
  • The ACS-VPS augmentation method alone improved the SELD score by 36.6%.
  • The TSL framework, when applied to the student model, resulted in a 59.2% improvement relative to the official baseline.
  • The VGDF method reduced the localization error (LECD) from 14.10° to 12.25°.
  • The authors' submission, using a model ensemble of the proposed techniques, secured first place in the DCASE 2023 Challenge SELD task.

Limitations and Future Work

  • Data limitations: The paper acknowledges the inherent challenge of limited audio-visual data, which is the primary motivation for their TSL approach. While their methods significantly mitigate this, further improvements might still be possible with more data.
  • Privacy concerns: The paper mentions that human faces are blurred in the dataset, limiting the direct use of facial features. This necessitates indirect cues like keypoints.
  • Future work: The authors suggest exploring more cross-modal fusion strategies and inter-modal relationships to learn useful cues embedded in multi-modality data. They also hint at investigating more advanced augmentation techniques.

Summary updated

Abstract

This study presents an audio-visual information fusion approach to sound event localization and detection (SELD) in low-resource scenarios. We aim at utilizing audio and video modality information through cross-modal learning and multi-modal fusion. First, we propose a cross-modal teacher-student learning (TSL) framework to transfer information from an audio-only teacher model, trained on a rich collection of audio data with multiple data augmentation techniques, to an audiovisual student model trained with only a limited set of multimodal data. Next, we propose a two-stage audio-visual fusion strategy, consisting of an early feature fusion and a late video-guided decision fusion to exploit synergies between audio and video modalities. Finally, we introduce an innovative video pixel swapping (VPS) technique to extend an audio channel swapping (ACS) method to an audio-visual joint augmentation. Evaluation results on the Detection and Classification of Acoustic Scenes and Events (DCASE) 2023 Challenge data set demonstrate significant improvements in SELD performances. Furthermore, our submission to the SELD task of the DCASE 2023 Challenge ranks first place by effectively integrating the proposed techniques into a model ensemble.
BibTeX
@inproceedings{jiang2024exploring,
  author = {Ya Jiang and Qing Wang and Jun Du and Maocheng Hu and Pengfei Hu and Zeyan Liu and Shi Cheng and Zhaoxu Nian and Yuxuan Dong and Mingqi Cai and Xin Fang and Chin-Hui Lee},
  title = {{Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection In Low-Resource Realistic Scenarios}},
  booktitle = {ICME},
  year = {2024},
  url = {https://openreview.net/forum?id=zpmYHrIIEB},
}
Figure