Abstract
Spatial localization of 3D sound sources is an important problem in many real-world scenarios, especially when the sources may not have any visually distinguishable characteristic; e.g., finding a gas leak or a malfunctioning motor. In this paper, we cast this task in a novel audio-visual setting by introducing an acoustic-camera rig consisting of a centered pinhole RGB camera and a uniform circular array of four coplanar microphones. Using this setup, we propose Sound3DVDet – a 3D sound source localization Transformer model that treats this task as a set prediction problem. It first learns a set of initial sound source locations (queries) from a single view of the microphone array signal, then feeds the query set to a sequence of Transformer-like layers for refinement. Each query arising from each layer repeatedly aggregates sound source cues from other views. We deeply supervise the initial sound source queries, intermediate layer queries, and the final output by measuring their respective discrepancy against ground truth queries via bipartite matching. To evaluate our method, we introduce the Sound3DVDet Dataset, consisting of nearly 6k scenes produced using the SoundSpaces simulator. We conduct extensive experiments and show the efficacy of our approach against closely related methods, demonstrating significant improvements in localization accuracy.
Introduction
Most existing audio-visual localization research assumes a strong correlation between visual and auditory cues, such as seeing a person speaking. However, in many industrial scenarios, sound sources are "visually uninformative"—for example, a high-pressure gas leak on a plain pipe or a malfunctioning internal component. While audio-only methods using microphone arrays provide a rough direction of arrival, they struggle with precise 3D positioning. Sound3DVDet bridges this gap by using multiview geometry and an "acoustic-camera" rig to pinpoint the exact 3D coordinates of multiple, potentially "invisible" sound sources.
Methodology
Sound3DVDet is a Transformer-based architecture that treats 3D sound detection as a set prediction task, inspired by the DETR object detection model.
Architecture and Query Refinement
The model processes four-channel audio from a single view using GCC-Phat (Generalized Cross-Correlation Phase Transform) maps to capture inter-channel phase differences. These provide a coarse estimate of the sound's spatial origin, serving as "initial queries." These queries are then refined through a sequence of Transformer layers that model dependencies between different sound sources and aggregate visual cues from multiple RGB views.
Visual "On-the-Surface" Constraint
A core innovation is the geometric constraint applied during refinement. The model decodes a query into a 3D position and projects it onto available RGB image planes. If the 3D point lies on a physical surface, its projections across different views should represent the same visual point. The model uses features from LoFTR (a pre-trained local feature matcher) to evaluate this consistency, effectively using vision to "snap" the audio estimate onto the nearest physical surface.
Dataset and Experiments
To evaluate the framework, the authors created the Sound3DVDet Dataset using SoundSpaces 2.0 and Matterport3D. It includes nearly 6,000 scenes with 1 to 10 sound sources per scene across six common indoor object categories (walls, chairs, tables, etc.). The audio component uses five distinct sound classes (sirens, alarms, etc.) sampled at 21kHz.
Results
Sound3DVDet significantly outperformed traditional audio-only baselines like SELDNet and SoundDoA across several metrics:
- mAP (Mean Average Precision): Achieved 0.308, more than doubling the performance of the best baseline (SoundDoA at 0.123).
- mAR (Mean Average Recall): Reached 0.998, suggesting the model is exceptionally good at finding all present sound sources.
- mALE (Mean Average Localization Error): Reduced the error to 0.588 meters, compared to 0.820 meters for the best baseline.
Key insights from the experiments revealed that LoFTR features are superior to standard ResNet features for this task because they can find correspondences even in texture-homogeneous regions like plain white walls. Furthermore, deep supervision of intermediate Transformer layers was found to be essential; removing it caused mAP to drop by over 50%.
Conclusion
Sound3DVDet demonstrates that combining multiview microphone arrays with RGB images can pinpoint sound sources that lack distinct visual features. While the current model assumes unoccluded paths and static scenes, it provides a robust foundation for future deployment on robotic platforms in complex real-world environments.
Summary updated