Abstract
LAVSS (Location-Guided Audio-Visual Spatial Audio Separation) is a novel framework designed to isolate individual sound sources from binaural mixtures by explicitly modeling the spatial location of objects. While traditional audio-visual separation methods primarily rely on visual appearance ("what"), LAVSS integrates 2D positional encodings and inter-microphone phase differences (IPD) to define "where" a sound originates. This dual-guidance system effectively disambiguates identical sources, such as two guitars, which are often indistinguishable by appearance alone. Evaluated on the FAIR-Play dataset, LAVSS achieves state-of-the-art performance with a Signal-to-Distortion Ratio (SDR) of 5.91 dB, demonstrating significant improvements in spatial audio separation for immersive applications like VR and AR.
Introduction
Traditional Audio-Visual Sound Separation (AVSS) systems typically operate on monaural (single-channel) audio, focusing on identifying instruments or speakers based on visual cues. However, these systems struggle in immersive environments where spatial properties are inherent. The core challenge is twofold: spatial ambiguity, where identical instruments cannot be distinguished by appearance, and the "where" gap, where models ignore the spatial timing differences in binaural audio.
LAVSS addresses these challenges by leveraging the natural correlation between an object's physical location in a video frame and the phase differences in the audio reaching the left and right "ears." By treating location as a first-class citizen, the model can separate sources that would baffle standard systems.
Methodology
The authors propose a multi-stage architecture that fuses visual, positional, and audio features to guide the separation process.
Vision and Position Networks
The model utilizes Faster R-CNN for object detection. Beyond extracting visual appearance features (the "what"), it employs a Position Network that uses sinusoidal positional encoding—a technique common in Transformers—to map the 2D coordinates of an object's bounding box into a high-dimensional embedding. This ensures that even if two objects look identical, their unique spatial coordinates provide a clear signal for disambiguation.
Audio Spatial Cues and Fusion
LAVSS incorporates Inter-microphone Phase Difference (IPD) alongside standard magnitude spectrograms. IPD captures the minute timing differences between the left and right audio channels, which is the primary mechanism humans use to localize sound. These features are processed through a specialized VP Cross-Attention module, which performs cross-modal attention between visual appearance and positional features, ensuring the model attends to the correct object at the correct location.
Transfer Learning
To overcome the scarcity of binaural training data, the model employs a monaural-to-binaural knowledge transfer strategy. It is first pre-trained on the massive MUSIC-21 monaural dataset to learn rich spectral characteristics of various instruments. It is then fine-tuned on the FAIR-Play binaural dataset, allowing the model to adapt its learned "what" knowledge to the spatial "where" task.
Key Results and Insights
LAVSS outperformed all existing benchmarks on the FAIR-Play dataset, achieving an average SDR of 5.91 dB and a Signal-to-Interference Ratio (SIR) of 10.19 dB. This represents a significant lead over the previous state-of-the-art, Sep-Stereo (5.26 dB SDR).
Same-Category Separation
The most compelling evidence of the model's efficacy is found in "same-category" separation tasks. When two identical instruments (e.g., two cellos) play simultaneously, LAVSS successfully isolates the signals, whereas competing models often produce blurred or mixed results. Qualitative analysis of predicted spectrograms confirms that LAVSS maintains higher signal integrity in these ambiguous scenarios.
Synergy of IPD and Position
Ablation studies revealed that IPD (audio location) and Positional Encoding (visual location) are highly complementary. Combining both cues resulted in a 1.42 dB SDR improvement over using either alone, suggesting that the model effectively synchronizes visual and auditory spatial information.
Limitations and Future Work
While LAVSS sets a new benchmark, it remains dependent on the accuracy of the initial object detector; if a source is not detected, the separation guidance fails. Future research may explore incorporating 3D depth information or 360° video to further refine spatial mapping, particularly in highly dynamic scenes with rapid movement.
Summary updated