Abstract
Sound event localization and detection (SELD) combines two subtasks: sound event detection (SED) and direction of arrival (DOA) estimation. While traditionally an audio-only problem, recent research has begun incorporating visual information. This paper explores the integration of audio and visual feature embeddings extracted with pre-trained deep networks, specifically testing ResNet50 and Inflated 3D ConvNet (I3D) for the visual modality. We compare two fusion methods: the AV-Conformer and the Cross-Modal Attentive Fusion (CMAF) model. Our best models significantly outperform the DCASE 2023 Task3 audio-only and AV baselines on the development set of the STARSS23 dataset, achieving competitive performance without model ensembling, heavy data augmentation, or prediction post-processing.
Introduction
Sound Event Localization and Detection (SELD) is a multi-faceted task requiring the identification of sound event classes and their spatial coordinates. While audio-only approaches are standard, they often struggle in noisy or acoustically complex environments. Visual data can provide critical spatial cues, yet previous multimodal efforts have largely relied on sparse information like bounding boxes or human pose keypoints. This work investigates the utility of dense visual embeddings—rich features from pre-trained networks—to enhance SELD performance, particularly when objects are visible but the acoustic environment is challenging.
Methodology
The proposed framework utilizes a dual-encoder architecture followed by an attention-based fusion module. The audio branch employs a CNN-Conformer processing 7-channel inputs, including 4-channel First-Order Ambisonics log-mel spectrograms and 3-channel Intensity Vectors. For the visual branch, pre-trained ResNet50 (for spatial features) or I3D (for temporal features) models extract embeddings from 360° video frames.
Two primary fusion strategies are evaluated:
- AV-Conformer: A straightforward approach where audio and visual embeddings are concatenated and passed through shared Conformer layers.
- Cross-Modal Attentive Fusion (CMAF): A more complex architecture utilizing parallel branches where modalities interact via Multi-Head Cross-Attention (MHCA).
The final output is represented as multi-channel multi-ACCDOA (m-ACCDOA) vectors, a representation that handles multiple simultaneous sound sources of the same class.
Experimental Setup
Evaluation was conducted on the STARSS23 (Sony-TAu Realistic Spatial Soundscapes 2023) dataset, which contains approximately 7.5 hours of real-world recordings. The dataset includes 13 event classes with up to 5 simultaneous events and directional interferers. The audio branch was pre-trained on 30 hours of simulated data from the DCASE 2022 challenge to provide a robust baseline.
Results and Discussion
The proposed AV models achieved a SELD score of 0.38 on the development set (where lower is better), a substantial improvement over the DCASE 2023 AV baseline (0.71) and audio-only baseline (0.48). Notably, the AV-Conformer reduced localization error (LE) to approximately 16°, compared to 22° for the audio-only model.
A key finding is that vision acts as a "spatial anchor," improving Localization Recall (LR) by roughly 4%. Interestingly, the simpler AV-Conformer generally outperformed the CMAF model. The CMAF architecture struggled with rare, short-duration classes like "Knock," suggesting that cross-attention requires more balanced data to learn effective inter-modal correlations.
Conclusion
Fusing dense visual embeddings with audio significantly enhances SELD performance. While audio remains the primary modality for detection, visual information provides vital spatial grounding for visible sources. Future work will explore visually guided post-processing—using object detectors to refine acoustic predictions—and more advanced data augmentation to further improve localization accuracy.
Summary updated