LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

Yuxin Ye, Wenming Yang, Yapeng Tian

Abstract

LAVSS (Location-Guided Audio-Visual Spatial Audio Separation) is a novel framework designed to isolate individual sound sources from binaural mixtures by explicitly modeling the spatial location of objects. While traditional audio-visual separation methods primarily rely on visual appearance ("what"), LAVSS integrates 2D positional encodings and inter-microphone phase differences (IPD) to define "where" a sound originates. This dual-guidance system effectively disambiguates identical sources, such as two guitars, which are often indistinguishable by appearance alone. Evaluated on the FAIR-Play dataset, LAVSS achieves state-of-the-art performance with a Signal-to-Distortion Ratio (SDR) of 5.91 dB, demonstrating significant improvements in spatial audio separation for immersive applications like VR and AR.

Introduction

Traditional Audio-Visual Sound Separation (AVSS) systems typically operate on monaural (single-channel) audio, focusing on identifying instruments or speakers based on visual cues. However, these systems struggle in immersive environments where spatial properties are inherent. The core challenge is twofold: spatial ambiguity, where identical instruments cannot be distinguished by appearance, and the "where" gap, where models ignore the spatial timing differences in binaural audio.

LAVSS addresses these challenges by leveraging the natural correlation between an object's physical location in a video frame and the phase differences in the audio reaching the left and right "ears." By treating location as a first-class citizen, the model can separate sources that would baffle standard systems.

Methodology

The authors propose a multi-stage architecture that fuses visual, positional, and audio features to guide the separation process.

Vision and Position Networks

The model utilizes Faster R-CNN for object detection. Beyond extracting visual appearance features (the "what"), it employs a Position Network that uses sinusoidal positional encoding—a technique common in Transformers—to map the 2D coordinates of an object's bounding box into a high-dimensional embedding. This ensures that even if two objects look identical, their unique spatial coordinates provide a clear signal for disambiguation.

Audio Spatial Cues and Fusion

LAVSS incorporates Inter-microphone Phase Difference (IPD) alongside standard magnitude spectrograms. IPD captures the minute timing differences between the left and right audio channels, which is the primary mechanism humans use to localize sound. These features are processed through a specialized VP Cross-Attention module, which performs cross-modal attention between visual appearance and positional features, ensuring the model attends to the correct object at the correct location.

The LAVSS architecture overview, illustrating the integration of Vision, Position, and Sound Separation Networks.

Transfer Learning

To overcome the scarcity of binaural training data, the model employs a monaural-to-binaural knowledge transfer strategy. It is first pre-trained on the massive MUSIC-21 monaural dataset to learn rich spectral characteristics of various instruments. It is then fine-tuned on the FAIR-Play binaural dataset, allowing the model to adapt its learned "what" knowledge to the spatial "where" task.

Key Results and Insights

LAVSS outperformed all existing benchmarks on the FAIR-Play dataset, achieving an average SDR of 5.91 dB and a Signal-to-Interference Ratio (SIR) of 10.19 dB. This represents a significant lead over the previous state-of-the-art, Sep-Stereo (5.26 dB SDR).

Same-Category Separation

The most compelling evidence of the model's efficacy is found in "same-category" separation tasks. When two identical instruments (e.g., two cellos) play simultaneously, LAVSS successfully isolates the signals, whereas competing models often produce blurred or mixed results. Qualitative analysis of predicted spectrograms confirms that LAVSS maintains higher signal integrity in these ambiguous scenarios.

Qualitative comparison of predicted spectrograms across different models for the FAIR-Play dataset, highlighting LAVSS's superior separation in same-category scenarios.

Synergy of IPD and Position

Ablation studies revealed that IPD (audio location) and Positional Encoding (visual location) are highly complementary. Combining both cues resulted in a 1.42 dB SDR improvement over using either alone, suggesting that the model effectively synchronizes visual and auditory spatial information.

Limitations and Future Work

While LAVSS sets a new benchmark, it remains dependent on the accuracy of the initial object detector; if a source is not detected, the separation guidance fails. Future research may explore incorporating 3D depth information or 360° video to further refine spatial mapping, particularly in highly dynamic scenes with rapid movement.

Summary updated

Abstract

Existing machine learning research has achieved promising results in monaural audio-visual separation (MAVS). However, most MAVS methods purely consider what the sound source is, not where it is located. This can be a problem in VR/AR scenarios, where listeners need to be able to distinguish between similar audio sources located in different directions. To address this limitation, we have generalized MAVS to spatial audio separation and proposed LAVSS: a location-guided audio-visual spatial audio separator. LAVSS is inspired by the correlation between spatial audio and visual location. We introduce the phase difference carried by binaural audio as spatial cues, and we utilize positional representations of sounding objects as additional modality guidance. We also leverage multi-level cross-modal attention to perform visual-positional collaboration with audio features. In addition, we adopt a pre-trained monaural separator to transfer knowledge from rich mono sounds to boost spatial audio separation. This exploits the correlation between monaural and binaural channels. Experiments on the FAIR-Play dataset demonstrate the superiority of the proposed LAVSS over existing benchmarks of audio-visual separation. Our project page: https://yyx666660.github.io/LAVSS/.
BibTeX
@inproceedings{ye2024lavss,
  author = {Yuxin Ye and Wenming Yang and Yapeng Tian},
  title = {{LAVSS: Location-Guided Audio-Visual Spatial Audio Separation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00542},
}
Figure