Audio-Visual Model Distillation Using Acoustic Images

Andrés F. Pérez, Valentina Sanguineti, Pietro Morerio, Vittorio Murino

Introduction

Traditional audio classification models typically process one-dimensional signals, such as raw waveforms or spectrograms, captured from a single microphone. While these models perform well in controlled environments, their performance often degrades significantly in real-world scenarios characterized by background noise, echoes, and overlapping sound sources. This paper, presented at WACV 2020, proposes a novel solution: leveraging spatialized "acoustic images" and visual data during training to distill more robust features into standard 1D audio models.

Methodology: The Teacher-Student Framework

The core of this approach is a knowledge distillation framework where high-capacity "teacher" models, trained on rich multimodal data, guide a "student" model that only has access to single-microphone audio at test time.

The DualCam Sensor

To generate the necessary training data, the researchers utilized a specialized hybrid sensor known as the "DualCam." This device integrates a standard optical camera with a 2D array of 128 microphones.

The DualCam hardware used for data collection, featuring a central camera surrounded by a 128-microphone array.

By applying beamforming techniques to the 128 audio streams, the system produces an acoustic image: a spatial map where each pixel's intensity corresponds to the sound energy originating from that direction. These acoustic images are perfectly aligned with the visual frames from the camera.

Distillation Process

The framework employs two types of teachers:

  1. DualCamNet: A model that processes acoustic image volumes (compressed using MFCCs).
  2. Visual Teacher: A ResNet-50 model that captures the dynamics of the action from RGB video.

The student model (either OurSoundNet for raw audio or HearNet for spectrograms) is trained using a specialized loss function. This function encourages the student to not only predict the correct class label but also to mimic the "soft" probability distributions of the teachers. This process transfers "dark knowledge"—spatial and visual context—into the 1D audio representation.

An example of an acoustic image showing the spatial distribution of sound intensity (red heatmap) during a hammering action.

Experimental Results

The researchers evaluated their method on a new Audio-Visually Indicated Action Dataset, which includes 14 action classes across three distinct environments: anechoic, indoor, and outdoor.

Key Findings

  • Superior Supervision: The acoustic image teacher (DualCamNet) consistently outperformed the visual teacher, achieving 84.7% accuracy in clean environments compared to ~71% for the RGB model. This indicates that spatial sound maps are often more discriminative for action recognition than video alone.
  • Robustness to Noise: Distillation significantly improved the student's ability to generalize across different environments. In cross-scenario tests, distillation from the DualCamNet teacher boosted the student's accuracy from 65.5% to 78.6%.
  • Feature Transferability: Representations learned through this distillation process were found to be effective on unrelated tasks, such as the DCASE-2018 acoustic scene classification benchmark, demonstrating that the model learns fundamentally better audio features.

Conclusion and Future Work

By using acoustic images as a supervisory signal, this work demonstrates that 1D audio models can inherit spatial awareness, making them far more robust to environmental noise. While the current requirement for specialized hardware like the DualCam is a hurdle for large-scale data collection, the results prove the value of cross-modal distillation. Future research may focus on self-supervised learning and real-time beamforming to scale this approach to larger, unlabelled datasets.

Summary updated

Abstract

In this paper, we investigate how to learn rich and robust feature representations for audio classification from visual data and acoustic images, a novel audio data modality. Former models learn audio representations from raw signals or spectral data acquired by a single microphone, with remarkable results in classification and retrieval. However, such representations are not so robust towards variable environmental sound conditions. We tackle this drawback by exploiting a new multimodal labeled action recognition dataset acquired by a hybrid audio-visual sensor that provides RGB video, raw audio signals, and spatialized acoustic data, also known as acoustic images, where the visual and acoustic images are aligned in space and synchronized in time. Using this richer information, we train audio deep learning models in a teacher-student fashion. In particular, we distill knowledge into audio networks from both visual and acoustic image teachers. Our experiments suggest that the learned representations are more powerful and have better generalization capabilities than the features learned from models trained using just single-microphone audio data.
BibTeX
@inproceedings{perez2020audio,
  author = {Andrés F. Pérez and Valentina Sanguineti and Pietro Morerio and Vittorio Murino},
  title = {{Audio-Visual Model Distillation Using Acoustic Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093307},
}
Figure