TL;DR
This paper presents a robust framework for Audio-Visual Sound Event Localization and Detection (AV-SELD) by fusing deep embeddings from both modalities. By combining a CNN-Conformer audio branch with pre-trained visual encoders (ResNet50 or I3D) via attention-based fusion, the authors achieved a significant performance leap over the DCASE 2023 baselines on the STARSS23 dataset.
What Problem Does This Paper Address?
Sound Event Localization and Detection (SELD) is the "double-threat" task of identifying what a sound is (detection) and where it is coming from (localization). Traditionally, this has been an audio-only problem. However, audio is prone to interference and struggles with precise spatial resolution in complex environments.
While researchers have recently started incorporating visual data, most previous attempts used "sparse" visual information, such as object bounding boxes or human pose keypoints. This paper addresses the gap in using dense visual embeddings—rich feature representations extracted directly from video frames—and explores how to best fuse them with acoustic data to improve localization accuracy, especially when objects are visible but the acoustic environment is noisy.
Methods
The authors propose a dual-encoder pipeline followed by an attention-based fusion module:
- Audio Encoder: Uses a CNN-Conformer architecture. It takes 7-channel inputs (4-channel First-Order Ambisonics log-mel spectrograms plus 3-channel Intensity Vectors) to capture both the "what" and the "where" of the sound.
- Visual Encoder: The team tested two heavyweights: ResNet50 (for spatial features) and I3D (Inflated 3D ConvNet, for temporal/motion features). They processed 360° video frames by splitting them into "left" and "right" views to maintain spatial resolution before embedding.
- Fusion Mechanisms: They compared two sophisticated approaches:
- AV-Conformer: Concatenates audio and visual embeddings and passes them through shared Conformer layers.
- Cross-Modal Attentive Fusion (CMAF): Uses parallel branches where one modality "queries" the other via Multi-Head Cross-Attention (MHCA).
- Output: The model predicts m-ACCDOA vectors, a representation that handles multiple simultaneous sound sources of the same class.
Datasets
The study primarily uses the STARSS23 (Sony-TAu Realistic Spatial Soundscapes 2023) dataset.
- Content: ~7.5 hours of real-world audio-visual recordings.
- Format: 360° video spatially and temporally aligned with tetrahedral microphone array audio.
- Complexity: Includes 13 event classes with up to 5 simultaneous events and directional interferers (noises that shouldn't be detected).
- Pre-training: The audio branch was pre-trained on ~30 hours of simulated data from the DCASE 2022 challenge.
Key Insights and Ideas
- Complementary Strengths: The authors confirm that audio is the primary driver for detection (SED), but vision acts as a powerful "spatial anchor." Vision provides high spatial accuracy for visible sources, while audio handles occluded objects that the camera can't see.
- Simplicity Wins: Surprisingly, the AV-Conformer (concatenation + attention) generally outperformed the more complex CMAF architecture. The authors noted that CMAF struggled significantly with rare, short-duration classes like "Knock," likely because the cross-attention mechanism didn't have enough data to learn the correlation for those specific events.
- The "Visual Boost": Adding visual embeddings improved Localization Recall (LR) by about 4% over audio-only models. This suggests that even when the audio is ambiguous, the visual presence of an object helps the model "recall" that a sound should be localized there.
Key Results
The proposed models crushed the existing benchmarks on the STARSS23 development set:
- SELD Score: The best AV model achieved a score of 0.38, a massive improvement over the DCASE 2023 Baseline AV (0.71) and Baseline Audio-only (0.48). (Note: Lower is better for SELD scores).
- Localization Error (LE): The AV-Conformer reduced spatial error to roughly 16°, compared to 22° for the audio-only baseline.
- Visual-Only Failure: A visual-only model performed poorly (SELD score of 0.87), proving that while vision helps, SELD remains fundamentally an acoustic task at its core.
Limitations and Future Work
- Data Imbalance: The failure of attention-based models to detect the "Knock" class (which appeared in only 9 seconds of training data) highlights a sensitivity to class imbalance in multimodal fusion.
- Static vs. Dynamic: While I3D was tested for temporal features, the authors found ResNet50 (2D) to be slightly more effective in some metrics, suggesting that motion cues might need more specialized handling in SELD.
- Future Directions: The authors suggest that visually guided post-processing—using an object detector to "snap" acoustic predictions to visible objects—could further refine localization. They also plan to explore heavier data augmentation and model ensembling to reach the very top of the DCASE leaderboards.
Summary updated