Audio-Visual Model Distillation Using Acoustic Images

Andrés F. Pérez, Valentina Sanguineti, Pietro Morerio, Vittorio Murino

Abstract

In this paper, we investigate how to learn rich and robust feature representations for audio classification from visual data and acoustic images, a novel audio data modality. Former models learn audio representations from raw signals or spectral data acquired by a single microphone, with remarkable results in classification and retrieval. However, such representations are not so robust towards variable environmental sound conditions. We tackle this drawback by exploiting a new multimodal labeled action recognition dataset acquired by a hybrid audio-visual sensor that provides RGB video, raw audio signals, and spatialized acoustic data, also known as acoustic images, where the visual and acoustic images are aligned in space and synchronized in time. Using this richer information, we train audio deep learning models in a teacher-student fashion. In particular, we distill knowledge into audio networks from both visual and acoustic image teachers. Our experiments suggest that the learned representations are more powerful and have better generalization capabilities than the features learned from models trained using just single-microphone audio data.

Introduction

Standard audio classification models typically learn from 1D raw signals or 2D spectrograms. While effective in controlled settings, these models often fail in the real world due to "domain shift"—changes in background noise, echoes, or room acoustics. Humans do not just hear sound; we perceive it as spatially localized events often tied to visual cues. Current machine learning models lack this spatial context. The challenge addressed here is how to give a standard, single-microphone system the spatial "wisdom" of a high-end sensor without requiring expensive hardware during deployment.

Methodology

The DualCam Sensor and Acoustic Images

The researchers utilized the "DualCam," a hybrid device featuring an optical camera and a 128-microphone array. This hardware allows for the creation of acoustic images, where sound intensity is mapped directly onto the pixels of a video frame, providing spatial context for audio events.

The DualCam hybrid sensor, featuring an optical camera and a 128-microphone array.

These acoustic images are processed as "multispectral acoustic image volumes"—essentially videos where each frame shows the spatial distribution of a specific audio frequency. This representation allows the model to see where sound originates across different frequency bands.

Acoustic images map sound intensity to spatial coordinates, forming a multispectral volume across frequencies.

Teacher-Student Distillation

The researchers employed a teacher-student distillation framework to transfer knowledge from complex modalities to a simple audio-only model:

  1. The Teachers: Two teacher networks were developed. DualCamNet is a 3D Convolutional Neural Network (CNN) that processes the multispectral acoustic image volumes. The Visual Teacher is a ResNet-50 trained on standard video frames to provide visual context.
  2. The Student: A 1D CNN (HearNet or OurSoundNet) that only receives raw audio from a single microphone.
  3. Distillation: During training, the student minimizes a loss function accounting for both ground-truth labels and the "soft labels" (probabilities) generated by the teachers. This transfers the spatial and visual intuition of the teachers into the student's weights.

The Audio-Visually Indicated Action Dataset

The authors introduced a new benchmark: the Audio-Visually Indicated Action Dataset. It contains 14 action classes (e.g., clapping, hammering, paper ripping) performed by 9 subjects. To test robustness, it was recorded in three distinct environments: an anechoic room (clean), an indoor open space (echoey), and a terrace (ambient outdoor noise). It is the first dataset to provide spatially aligned RGB video, raw audio, and multispectral acoustic images.

Key Results and Insights

  • Accuracy Boost: On the new dataset, the HearNet student improved its accuracy from 65.5% (trained on audio only) to 78.6% when distilled from the acoustic image teacher.
  • Teacher Performance: The DualCamNet teacher achieved up to 84.7% accuracy, proving that acoustic images are a highly discriminative modality for action recognition.
  • Cross-Scenario Robustness: Distillation significantly improved generalization. A student distilled from an acoustic teacher in a clean room performed much better when tested in a noisy outdoor environment compared to a student trained only on clean audio.
  • Acoustic Images vs. Video: Interestingly, the acoustic image teacher often outperformed the visual teacher. Many actions (like "clicking" or "snapping fingers") involve small movements that are hard to see but produce distinct, localized sound patterns.

Conclusion

By aligning sound to specific spatial coordinates, the proposed method provides a cleaner learning signal than mixed 1D waveforms, helping models disentangle sounds of interest from background clutter. While the method currently relies on specialized hardware for training data collection, the resulting student models offer significantly improved performance using only standard single-microphone input. Future work will explore scaling the dataset and applying acoustic images to sound source localization in complex environments.

Summary updated

Abstract

In this paper, we investigate how to learn rich and robust feature representations for audio classification from visual data and acoustic images, a novel audio data modality. Former models learn audio representations from raw signals or spectral data acquired by a single microphone, with remarkable results in classification and retrieval. However, such representations are not so robust towards variable environmental sound conditions. We tackle this drawback by exploiting a new multimodal labeled action recognition dataset acquired by a hybrid audio-visual sensor that provides RGB video, raw audio signals, and spatialized acoustic data, also known as acoustic images, where the visual and acoustic images are aligned in space and synchronized in time. Using this richer information, we train audio deep learning models in a teacher-student fashion. In particular, we distill knowledge into audio networks from both visual and acoustic image teachers. Our experiments suggest that the learned representations are more powerful and have better generalization capabilities than the features learned from models trained using just single-microphone audio data.
BibTeX
@inproceedings{perez2020audio,
  author = {Andrés F. Pérez and Valentina Sanguineti and Pietro Morerio and Vittorio Murino},
  title = {{Audio-Visual Model Distillation Using Acoustic Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093307},
}
Figure