Introduction
Traditional audio classification models typically process one-dimensional signals, such as raw waveforms or spectrograms, captured from a single microphone. While these models perform well in controlled environments, their performance often degrades significantly in real-world scenarios characterized by background noise, echoes, and overlapping sound sources. This paper, presented at WACV 2020, proposes a novel solution: leveraging spatialized "acoustic images" and visual data during training to distill more robust features into standard 1D audio models.
Methodology: The Teacher-Student Framework
The core of this approach is a knowledge distillation framework where high-capacity "teacher" models, trained on rich multimodal data, guide a "student" model that only has access to single-microphone audio at test time.
The DualCam Sensor
To generate the necessary training data, the researchers utilized a specialized hybrid sensor known as the "DualCam." This device integrates a standard optical camera with a 2D array of 128 microphones.
By applying beamforming techniques to the 128 audio streams, the system produces an acoustic image: a spatial map where each pixel's intensity corresponds to the sound energy originating from that direction. These acoustic images are perfectly aligned with the visual frames from the camera.
Distillation Process
The framework employs two types of teachers:
- DualCamNet: A model that processes acoustic image volumes (compressed using MFCCs).
- Visual Teacher: A ResNet-50 model that captures the dynamics of the action from RGB video.
The student model (either OurSoundNet for raw audio or HearNet for spectrograms) is trained using a specialized loss function. This function encourages the student to not only predict the correct class label but also to mimic the "soft" probability distributions of the teachers. This process transfers "dark knowledge"—spatial and visual context—into the 1D audio representation.
Experimental Results
The researchers evaluated their method on a new Audio-Visually Indicated Action Dataset, which includes 14 action classes across three distinct environments: anechoic, indoor, and outdoor.
Key Findings
- Superior Supervision: The acoustic image teacher (DualCamNet) consistently outperformed the visual teacher, achieving 84.7% accuracy in clean environments compared to ~71% for the RGB model. This indicates that spatial sound maps are often more discriminative for action recognition than video alone.
- Robustness to Noise: Distillation significantly improved the student's ability to generalize across different environments. In cross-scenario tests, distillation from the DualCamNet teacher boosted the student's accuracy from 65.5% to 78.6%.
- Feature Transferability: Representations learned through this distillation process were found to be effective on unrelated tasks, such as the DCASE-2018 acoustic scene classification benchmark, demonstrating that the model learns fundamentally better audio features.
Conclusion and Future Work
By using acoustic images as a supervisory signal, this work demonstrates that 1D audio models can inherit spatial awareness, making them far more robust to environmental noise. While the current requirement for specialized hardware like the DualCam is a hurdle for large-scale data collection, the results prove the value of cross-modal distillation. Future research may focus on self-supervised learning and real-time beamforming to scale this approach to larger, unlabelled datasets.
Summary updated