LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

Yuxin Ye, Wenming Yang, Yapeng Tian

Abstract: Existing machine learning research has achieved promising results in monaural audio-visual separation (MAVS). However, most MAVS methods purely consider what the sound source is, not where it is located. This can be a problem in VR/AR scenarios, where listeners need to be able to distinguish between similar audio sources located in different directions. To address this limitation, we have generalized MAVS to spatial audio separation and proposed LAVSS: a location-guided audio-visual spatial audio separator. LAVSS is inspired by the correlation between spatial audio and visual location. We introduce the phase difference carried by binaural audio as spatial cues, and we utilize positional representations of sounding objects as additional modality guidance. We also leverage multi-level cross-modal attention to perform visual-positional collaboration with audio features. In addition, we adopt a pre-trained monaural separator to transfer knowledge from rich mono sounds to boost spatial audio separation. This exploits the correlation between monaural and binaural channels. Experiments on the FAIR-Play dataset demonstrate the superiority of the proposed LAVSS over existing benchmarks of audio-visual separation.

Introduction

Traditional Audio-Visual Separation (AVS) systems are adept at identifying "what" is making a sound (e.g., a piano vs. a guitar) but often struggle in spatial environments like Virtual or Augmented Reality (VR/AR). In a 3D scene, two identical instruments might play simultaneously from different locations. A standard monaural system often fails to distinguish between them because their visual signatures are too similar.

LAVSS (Location-Guided Audio-Visual Spatial Audio Separation) addresses this "spatial gap" by moving from Monaural AVS (MAVS) to Audio-Visual Spatial Audio Separation (AVSS). The goal is to recover distinct left-and-right ear signals for specific sources based on their physical location in the video frame, leveraging the inherent correlation between spatial audio and visual position.

Methodology

The LAVSS framework employs a multi-pronged approach to fuse visual, positional, and auditory cues into a cohesive separation model.

  1. Vision-Position Embedding: The system detects objects using Faster R-CNN and extracts regional visual features via ResNet-18. To capture "where" an object is, the authors introduce explicit 2D Positional Encoding. This maps the $(x, y)$ coordinates of the sounding object into a high-dimensional space, forcing the network to learn the relationship between screen position and spatial audio profiles.
  2. Spatial Audio Cues (IPD): To assist the model in understanding directionality, the framework calculates the Inter-microphone Phase Difference (IPD). This measures the sub-millisecond delay between sound hitting the left versus right microphone, a primary cue for human localization.
  3. Multi-Scale Cross-Modal Attention: The model utilizes Cross-Modal Attention (CMA) blocks. These allow visual features to query positional features and vice versa, ensuring the audio separator accurately maps visual regions to their corresponding spatial audio cues.
  4. Transfer Learning: Due to the limited size of binaural datasets, the separator is first pre-trained on the massive MUSIC-21 monaural dataset. This knowledge of instrument sounds is then transferred to the spatial task, bridging the data scarcity gap.
The LAVSS architecture integrates Vision, Position, and Sound Separation networks to perform location-guided audio separation.

Experimental Results

LAVSS was evaluated on the FAIR-Play dataset, which consists of 1,915 videos recorded with a professional binaural microphone setup simulating human hearing.

Quantitative Performance

LAVSS outperformed existing state-of-the-art (SOTA) methods, including Sep-Stereo and CCoL, across standard metrics:

  • Signal-to-Distortion Ratio (SDR): Achieved 5.91 dB, a significant improvement over the previous best of 5.26 dB.
  • Signal-to-Interference Ratio (SIR): Reached 10.19 dB, outperforming the closest competitor (8.96 dB) by a wide margin.

Qualitative Analysis

In the most difficult scenarios—such as separating two instruments of the same type (e.g., two guitars)—LAVSS demonstrated a dramatic qualitative lead. While baseline models often produced mixed signals or "ghosting," LAVSS successfully isolated sounds based on their distinct spatial locations.

Qualitative comparison of predicted spectrograms for Cello and Guitar. LAVSS shows clearer separation and fewer artifacts compared to 2.5D Sep and SOPs.

Conclusion and Future Work

The primary innovation of LAVSS is treating an object's spatial coordinates as a first-class modality. By encoding position explicitly and leveraging IPD, the model creates a synergy between visual location and auditory phase shifts.

Future research will focus on expanding the framework to dynamic scenes with fast-moving objects or camera movement. Additionally, reducing the computational complexity of the multi-scale attention mechanism will be critical for deploying these models in real-time mobile VR/AR applications.

Summary updated

Abstract

Existing machine learning research has achieved promising results in monaural audio-visual separation (MAVS). However, most MAVS methods purely consider what the sound source is, not where it is located. This can be a problem in VR/AR scenarios, where listeners need to be able to distinguish between similar audio sources located in different directions. To address this limitation, we have generalized MAVS to spatial audio separation and proposed LAVSS: a location-guided audio-visual spatial audio separator. LAVSS is inspired by the correlation between spatial audio and visual location. We introduce the phase difference carried by binaural audio as spatial cues, and we utilize positional representations of sounding objects as additional modality guidance. We also leverage multi-level cross-modal attention to perform visual-positional collaboration with audio features. In addition, we adopt a pre-trained monaural separator to transfer knowledge from rich mono sounds to boost spatial audio separation. This exploits the correlation between monaural and binaural channels. Experiments on the FAIR-Play dataset demonstrate the superiority of the proposed LAVSS over existing benchmarks of audio-visual separation. Our project page: https://yyx666660.github.io/LAVSS/.
BibTeX
@inproceedings{ye2024lavss,
  author = {Yuxin Ye and Wenming Yang and Yapeng Tian},
  title = {{LAVSS: Location-Guided Audio-Visual Spatial Audio Separation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00542},
}
Figure