Sound3DVDet: 3D Sound Source Detection using Multiview Microphone Array and RGB Images

Yuhang He, Sangyun Shin, Anoop Cherian, Niki Trigoni, Andrew MarkhamWACV 2024 · OpenReview

Abstract

Spatial localization of 3D sound sources is an important problem in many real-world scenarios, especially when the sources may not have any visually distinguishable characteristic; e.g., finding a gas leak or a malfunctioning motor. In this paper, we cast this task in a novel audio-visual setting by introducing an acoustic-camera rig consisting of a centered pinhole RGB camera and a uniform circular array of four coplanar microphones. Using this setup, we propose Sound3DVDet – a 3D sound source localization Transformer model that treats this task as a set prediction problem. It first learns a set of initial sound source locations (queries) from a single view of the microphone array signal, then feeds the query set to a sequence of Transformer-like layers for refinement. Each query arising from each layer repeatedly aggregates sound source cues from other views. We deeply supervise the initial sound source queries, intermediate layer queries, and the final output by measuring their respective discrepancy against ground truth queries via bipartite matching. To evaluate our method, we introduce the Sound3DVDet Dataset, consisting of nearly 6k scenes produced using the SoundSpaces simulator. We conduct extensive experiments and show the efficacy of our approach against closely related methods, demonstrating significant improvements in localization accuracy.

Introduction

Most existing audio-visual localization research assumes a strong correlation between visual and auditory cues, such as seeing a person speaking. However, in many industrial scenarios, sound sources are "visually uninformative"—for example, a high-pressure gas leak on a plain pipe or a malfunctioning internal component. While audio-only methods using microphone arrays provide a rough direction of arrival, they struggle with precise 3D positioning. Sound3DVDet bridges this gap by using multiview geometry and an "acoustic-camera" rig to pinpoint the exact 3D coordinates of multiple, potentially "invisible" sound sources.

Methodology

Sound3DVDet is a Transformer-based architecture that treats 3D sound detection as a set prediction task, inspired by the DETR object detection model.

Architecture and Query Refinement

The model processes four-channel audio from a single view using GCC-Phat (Generalized Cross-Correlation Phase Transform) maps to capture inter-channel phase differences. These provide a coarse estimate of the sound's spatial origin, serving as "initial queries." These queries are then refined through a sequence of Transformer layers that model dependencies between different sound sources and aggregate visual cues from multiple RGB views.

The Sound3DVDet architecture featuring a detection backbone for query refinement and a visual feature aggregation module for multiview fusion.

Visual "On-the-Surface" Constraint

A core innovation is the geometric constraint applied during refinement. The model decodes a query into a 3D position and projects it onto available RGB image planes. If the 3D point lies on a physical surface, its projections across different views should represent the same visual point. The model uses features from LoFTR (a pre-trained local feature matcher) to evaluate this consistency, effectively using vision to "snap" the audio estimate onto the nearest physical surface.

Dataset and Experiments

To evaluate the framework, the authors created the Sound3DVDet Dataset using SoundSpaces 2.0 and Matterport3D. It includes nearly 6,000 scenes with 1 to 10 sound sources per scene across six common indoor object categories (walls, chairs, tables, etc.). The audio component uses five distinct sound classes (sirens, alarms, etc.) sampled at 21kHz.

Results

Sound3DVDet significantly outperformed traditional audio-only baselines like SELDNet and SoundDoA across several metrics:

  • mAP (Mean Average Precision): Achieved 0.308, more than doubling the performance of the best baseline (SoundDoA at 0.123).
  • mAR (Mean Average Recall): Reached 0.998, suggesting the model is exceptionally good at finding all present sound sources.
  • mALE (Mean Average Localization Error): Reduced the error to 0.588 meters, compared to 0.820 meters for the best baseline.
Qualitative results demonstrating Sound3DVDet's ability to localize sound sources across various indoor objects like cabinets, doors, and furniture.

Key insights from the experiments revealed that LoFTR features are superior to standard ResNet features for this task because they can find correspondences even in texture-homogeneous regions like plain white walls. Furthermore, deep supervision of intermediate Transformer layers was found to be essential; removing it caused mAP to drop by over 50%.

Conclusion

Sound3DVDet demonstrates that combining multiview microphone arrays with RGB images can pinpoint sound sources that lack distinct visual features. While the current model assumes unoccluded paths and static scenes, it provides a robust foundation for future deployment on robotic platforms in complex real-world environments.

Summary updated

Abstract

Spatial localization of 3D sound sources is an important problem in many real world scenarios, especially when the sources may not have any visually distinguishable characteristic; e.g., finding a gas leak, a malfunctioning motor, etc. In this paper, we cast this task in a novel audio-visual setting, by introducing an acoustic-camera rig consisting of a centered pinhole RGB camera and a uniform circular array of four coplanar microphones. Using this setup, we propose Sound3DVDet – a 3D sound source localization Transformer model that treats this task as a set prediction problem. It first learns a set of initial sound source locations (dubbed queries) from a single view of the microphone array signal, then feeds the query set to a sequence of Transformerlike layers for refinement. Each query arising from each layer repeatedly aggregates sound source cues from other views. We deeply supervise the initial sound source queries, intermediate layer queries, and the final output by measuring their respective discrepancy against ground truth queries via bipartite matching. To evaluate our method, we introduce a new dataset: Sound3DVDet Dataset, consisting of nearly 6k scenes produced using the SoundSpaces simulator. We conduct extensive experiments on our dataset and show the efficacy of our approach against closely related methods, demonstrating significant improvements in the localization accuracy. Code is available at https://github.com/yuhanghe01/Sound3DVDet.
BibTeX
@inproceedings{he2024sound,
  author = {Yuhang He and Sangyun Shin and Anoop Cherian and Niki Trigoni and Andrew Markham},
  title = {{Sound3DVDet: 3D Sound Source Detection using Multiview Microphone Array and RGB Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=1Rh1gtbOc3},
}
Figure