Introduction
While computer vision has achieved remarkable success in object recognition, auditory perception remains a relatively untapped frontier for spatial reasoning. Humans naturally integrate sight and sound to navigate, yet most AI systems rely exclusively on pixels. This research introduces a multi-task learning framework that enables machines to "see" using only binaural audio. By processing two-channel sound, the system performs dense semantic labeling of sound-making objects, predicts scene depth, and executes Spatial Sound Super-Resolution (S3R).
Methodology: Cross-Modal Distillation
To overcome the difficulty of manually labeling audio for semantic segments, the authors employ a cross-modal distillation strategy using a Teacher-Student framework.
- The Teacher (Vision): Pre-trained models (DeepLabv3+ and MonoDepth2) process 360° video frames to generate "pseudo-labels" for semantics and depth.
- The Student (Audio): A deep neural network is trained to predict these visual labels using only synchronized binaural audio as input.
Multi-Task Architecture
The network utilizes a shared encoder with Atrous Spatial Pyramid Pooling (ASPP) and three task-specific decoders:
- Semantic Prediction: Identifying objects like cars, trams, and motorcycles in a 360° field of view.
- Depth Estimation: Predicting the distance of objects based on acoustic cues.
- Spatial Sound Super-Resolution (S3R): A novel auxiliary task where the model predicts how audio would sound if the sensor were rotated. This mimics human head movement to resolve spatial ambiguities.
The Omni Auditory Perception Dataset
Because no existing dataset provided 360° coverage for both audio and video, the researchers developed the Omni Auditory Perception Dataset.
- Hardware: A custom rig featuring a 360° GoPro Fusion camera and a 3Dio Omni Binaural Microphone with four pairs of simulated human ears.
- Scale: 15 hours of data across 165 urban locations, totaling 64,250 clips.
- Processing: A majority-voting technique was used for background subtraction to ensure the model focuses on moving, sound-making objects rather than stationary ones.
Key Insights and Results
The study yields several critical findings regarding auditory spatial reasoning:
- Binaural Superiority: Binaural audio significantly outperforms monaural setups. The Interaural Time Difference (ITD) and Interaural Level Difference (ILD) are essential for accurate spatial localization. The full multi-task model achieved a 43.40% mIoU on semantic labeling, nearly double the 22.12% achieved by a monaural setup.
- Synergistic Multi-Tasking: Training for depth and S3R simultaneously improves semantic labeling. Adding the S3R task alone boosted semantic mIoU from 32.01% to 42.64%.
- Visual Robustness: The system remains functional in scenarios where cameras typically fail, such as heavy fog, rain, or nighttime conditions.
Limitations and Future Directions
Currently, the model's performance is capped by the accuracy of its vision-based "teachers." Future work aims to incorporate 3D LiDAR for more precise ground-truth depth data. Additionally, expanding the dataset beyond three object classes (car, tram, motorcycle) to include pedestrians and sirens will be necessary for broader real-world application. While the current semantic masks are coarser than vision-based systems, this work proves that sound alone can provide a sophisticated spatial understanding of complex environments.
Summary updated