Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection

Davide Berghi, Peipei Wu, Jinzheng Zhao, Wenwu Wang, Philip J. B. Jackson

TL;DR

This paper presents a robust framework for Audio-Visual Sound Event Localization and Detection (AV-SELD) by fusing deep embeddings from both modalities. By combining a CNN-Conformer audio branch with pre-trained visual encoders (ResNet50 or I3D) via attention-based fusion, the authors achieved a significant performance leap over the DCASE 2023 baselines on the STARSS23 dataset.

What Problem Does This Paper Address?

Sound Event Localization and Detection (SELD) is the "double-threat" task of identifying what a sound is (detection) and where it is coming from (localization). Traditionally, this has been an audio-only problem. However, audio is prone to interference and struggles with precise spatial resolution in complex environments.

While researchers have recently started incorporating visual data, most previous attempts used "sparse" visual information, such as object bounding boxes or human pose keypoints. This paper addresses the gap in using dense visual embeddings—rich feature representations extracted directly from video frames—and explores how to best fuse them with acoustic data to improve localization accuracy, especially when objects are visible but the acoustic environment is noisy.

Methods

The authors propose a dual-encoder pipeline followed by an attention-based fusion module:

  1. Audio Encoder: Uses a CNN-Conformer architecture. It takes 7-channel inputs (4-channel First-Order Ambisonics log-mel spectrograms plus 3-channel Intensity Vectors) to capture both the "what" and the "where" of the sound.
  2. Visual Encoder: The team tested two heavyweights: ResNet50 (for spatial features) and I3D (Inflated 3D ConvNet, for temporal/motion features). They processed 360° video frames by splitting them into "left" and "right" views to maintain spatial resolution before embedding.
  3. Fusion Mechanisms: They compared two sophisticated approaches:
  4. AV-Conformer: Concatenates audio and visual embeddings and passes them through shared Conformer layers.
  5. Cross-Modal Attentive Fusion (CMAF): Uses parallel branches where one modality "queries" the other via Multi-Head Cross-Attention (MHCA).
  6. Output: The model predicts m-ACCDOA vectors, a representation that handles multiple simultaneous sound sources of the same class.

Datasets

The study primarily uses the STARSS23 (Sony-TAu Realistic Spatial Soundscapes 2023) dataset.

  • Content: ~7.5 hours of real-world audio-visual recordings.
  • Format: 360° video spatially and temporally aligned with tetrahedral microphone array audio.
  • Complexity: Includes 13 event classes with up to 5 simultaneous events and directional interferers (noises that shouldn't be detected).
  • Pre-training: The audio branch was pre-trained on ~30 hours of simulated data from the DCASE 2022 challenge.

Key Insights and Ideas

  • Complementary Strengths: The authors confirm that audio is the primary driver for detection (SED), but vision acts as a powerful "spatial anchor." Vision provides high spatial accuracy for visible sources, while audio handles occluded objects that the camera can't see.
  • Simplicity Wins: Surprisingly, the AV-Conformer (concatenation + attention) generally outperformed the more complex CMAF architecture. The authors noted that CMAF struggled significantly with rare, short-duration classes like "Knock," likely because the cross-attention mechanism didn't have enough data to learn the correlation for those specific events.
  • The "Visual Boost": Adding visual embeddings improved Localization Recall (LR) by about 4% over audio-only models. This suggests that even when the audio is ambiguous, the visual presence of an object helps the model "recall" that a sound should be localized there.

Key Results

The proposed models crushed the existing benchmarks on the STARSS23 development set:

  • SELD Score: The best AV model achieved a score of 0.38, a massive improvement over the DCASE 2023 Baseline AV (0.71) and Baseline Audio-only (0.48). (Note: Lower is better for SELD scores).
  • Localization Error (LE): The AV-Conformer reduced spatial error to roughly 16°, compared to 22° for the audio-only baseline.
  • Visual-Only Failure: A visual-only model performed poorly (SELD score of 0.87), proving that while vision helps, SELD remains fundamentally an acoustic task at its core.

Limitations and Future Work

  • Data Imbalance: The failure of attention-based models to detect the "Knock" class (which appeared in only 9 seconds of training data) highlights a sensitivity to class imbalance in multimodal fusion.
  • Static vs. Dynamic: While I3D was tested for temporal features, the authors found ResNet50 (2D) to be slightly more effective in some metrics, suggesting that motion cues might need more specialized handling in SELD.
  • Future Directions: The authors suggest that visually guided post-processing—using an object detector to "snap" acoustic predictions to visible objects—could further refine localization. They also plan to explore heavier data augmentation and model ensembling to reach the very top of the DCASE leaderboards.

Summary updated

Abstract

Sound event localization and detection (SELD) combines two subtasks: sound event detection (SED) and direction of arrival (DOA) estimation. SELD is usually tackled as an audio-only problem, but visual information has been recently included. Few audio-visual (AV)-SELD works have been published and most employ vision via face/object bounding boxes, or human pose keypoints. In contrast, we explore the integration of audio and visual feature embeddings extracted with pre-trained deep networks. For the visual modality, we tested ResNet50 and Inflated 3D ConvNet (I3D). Our comparison of AV fusion methods includes the AV-Conformer and Cross-Modal Attentive Fusion (CMAF) model. Our best models outperform the DCASE 2023 Task3 audio-only and AV baselines by a wide margin on the development set of the STARSS23 dataset, making them competitive amongst state-of-the-art results of the AV challenge, without model ensembling, heavy data augmentation, or prediction post-processing. Such techniques and further pre-training could be applied as next steps to improve performance.
BibTeX
@inproceedings{berghi2024fusion,
  author = {Davide Berghi and Peipei Wu and Jinzheng Zhao and Wenwu Wang and Philip J. B. Jackson},
  title = {{Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=zgpg04r9pm},
}

Loading related content...

Figure