Binaural Audio-Visual Localization

Xinyi Wu, Zhenyao Wu, Lili Ju, Song WangAAAI 2021 · DOI

Abstract

This paper introduces BAVNet (Binaural Audio-Visual Network), a deep learning framework designed for pixel-level sound source localization using binaural audio. Unlike traditional monaural approaches that lack spatial depth, BAVNet leverages Interaural Time Differences (ITD) and Interaural Level Differences (ILD) inherent in two-channel audio. By employing a novel horizontal symmetry training strategy and temporal modeling via ConvLSTMs, the model achieves superior precision in identifying sound-producing regions, even in visually challenging scenarios. Experimental results on the FAIR-Play and YT-Music datasets demonstrate that BAVNet significantly outperforms state-of-the-art monaural and video-only baselines.

Introduction

Audio-visual localization is the task of identifying the visual source of a sound within a video frame. While significant progress has been made, existing research faces two primary limitations. First, most models rely on monaural audio, which lacks the spatial cues humans use to perceive directionality. Without these cues, AI models often over-rely on visual motion patterns rather than true auditory localization. Second, localization is frequently treated as an object-level task using coarse bounding boxes. This approach is imprecise; for instance, a sound does not originate from a "box" containing a musician, but specifically from the instrument's strings or the singer's mouth.

To address these gaps, we propose a transition toward pixel-level localization using binaural audio, mimicking the precision of human audio-visual integration. Our approach, BAVNet, utilizes spatial audio to pinpoint sound sources at a much finer resolution than previously possible.

Methods

BAVNet Architecture

BAVNet is a multi-stream architecture designed to fuse sight and sound through several specialized components:

  • Visual Stream: A VGG19-based encoder extracts multi-level features from raw video frames.
  • Binaural Audio Stream: Two-channel audio is processed into spectrograms using Short-time Fourier Transforms (STFT), preserving the phase and intensity differences between channels.
  • Temporal Modeling: The network employs ConvLSTMs (Convolutional Long Short-Term Memory) to capture the temporal evolution of both sound and visual scenes, which is essential for tracking moving sound sources.
  • Pixel-Level Decoder: Instead of predicting coordinates for a bounding box, a decoder produces a continuous heat map (localization map) identifying the exact pixels responsible for the sound.
The BAVNet architecture, featuring multi-stream encoders for visual and binaural audio data, integrated via ConvLSTMs and a horizontal flipping strategy for spatial learning.

The Horizontal Symmetry Strategy

A key innovation in BAVNet is the "flipping" operation used during training. To help the network learn spatial relationships, we map the left audio channel to the original video frame and the right audio channel to a horizontally flipped version of that frame. This symmetry trick forces the network to explicitly learn the mapping between specific audio intensities and their corresponding spatial locations in the visual field, effectively teaching the model "where" a sound is coming from based on channel imbalances.

Experimental Setup

We evaluated BAVNet on two primary datasets, manually enhanced with point-annotations to provide pixel-level ground truth:

  1. FAIR-Play: 1,871 video clips recorded in a music room using professional binaural microphones.
  2. YT-Music: 317 YouTube musical performances. To simulate binaural audio for this dataset, we applied a Head-Related Transfer Function (HRTF) to convert standard recordings into a binaural format.

Given that our model produces heatmaps, we utilized evaluation metrics from visual saliency research, including Pearson’s Correlation Coefficient (CC), Similarity Metric (SIM), and Earth Mover’s Distance (EMD), providing a more nuanced accuracy assessment than traditional Intersection over Union (IoU) metrics.

Results and Discussion

BAVNet consistently outperformed all baselines, including monaural-visual and video-only models. On the FAIR-Play dataset, BAVNet achieved a Pearson’s Correlation Coefficient (CC) of 0.776, compared to 0.743 for monaural audio and 0.679 for video-only approaches.

Qualitative comparison of sound source localization: (a) original frame, (b) monaural-visual localization baseline, and (c) BAVNet's pixel-level binaural-visual localization.

Qualitative visualizations (as seen in Figure 6) demonstrate that BAVNet accurately identifies specific sound-producing regions, such as a singer's throat or the specific hand position on a piano. In contrast, monaural models often "guessed" the center of the largest moving object. Furthermore, binaural audio acts as a critical fail-safe; in low-visibility conditions (e.g., night or fog), the spatial auditory cues allow BAVNet to maintain high localization accuracy where visual-only models fail.

Conclusion and Future Work

BAVNet demonstrates that binaural audio is a game-changer for audio-visual tasks, providing the spatial "where" that monaural audio lacks. While the current point-annotation strategy is effective, it is labor-intensive. Future work will explore more automated grounding techniques and test the model in complex "cocktail party" scenarios involving multiple overlapping, non-musical sound sources in reverberant environments.

Summary updated

BibTeX
@inproceedings{wu2021binaural,
  author = {Xinyi Wu and Zhenyao Wu and Lili Ju and Song Wang},
  title = {{Binaural Audio-Visual Localization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16403},
}
Figure