Self-supervised Audio Spatialization with Correspondence Classifier

Yu-Ding Lu, Hsin-Ying Lee, Hung-Yu Tseng, Ming-Hsuan Yang

Abstract: Spatial audio is an essential medium to audiences for 3D visual and auditory experience. However, the recording devices and techniques are expensive or inaccessible to the general public. In this work, we propose a self-supervised audio spatialization network that can generate spatial audio given the corresponding video and monaural audio. To enhance spatialization performance, we use an auxiliary classifier to classify ground-truth videos and those with audio where the left and right channels are swapped. We collect a large-scale video dataset with spatial audio to validate the proposed method. Experimental results demonstrate the effectiveness of the proposed model on the audio spatialization task.

Introduction

While 3D visual experiences are becoming common, spatial audio—which allows listeners to pinpoint the direction of a sound source—remains difficult to produce. Recording binaural audio typically requires expensive specialized equipment, such as "dummy head" microphones, that the general public rarely owns. Consequently, most user-generated content is limited to monaural (mono) audio, which lacks a sense of spatial immersion.

Existing research has explored "upconverting" mono to spatial audio, but these efforts often focus on $360^\circ$ video formats. This paper addresses the challenge of audio spatialization for general, "in-the-wild" videos by leveraging the natural correlation between visual motion (e.g., a car moving across the frame) and auditory cues.

Methodology

The proposed Audio Spatialization Network (ASN) consists of two primary components designed to work in tandem: a Spatial Audio Synthesizer and a Correspondence Classifier.

Spatial Audio Synthesizer

The synthesizer utilizes a U-Net-based architecture to process three distinct inputs: monaural audio (transformed into a spectrogram via Short-Time Fourier Transform), RGB frames to capture object appearance, and optical flow to capture object motion. Instead of predicting raw waveforms, the synthesizer predicts Ideal Ratio Masks (IRM) for the left and right channels. These masks are applied to the original mono audio to effectively "steer" the sound to the correct spatial orientation.

Correspondence Classifier

The "self-supervised" core of the framework is the Correspondence Classifier. Inspired by ACGANs, this binary classifier is trained to distinguish between a "correct" video-audio pair and a "flipped" pair where the left and right audio channels have been swapped. This mechanism forces the synthesizer to produce masks that are not just acoustically plausible, but spatially consistent with the visual movement on screen.

The Audio Spatialization Network (ASN) architecture, featuring the Spatial Audio Synthesizer and the auxiliary Correspondence Classifier.

Datasets and Evaluation

Due to the scarcity of standard datasets for this task, the authors curated a new large-scale dataset from YouTube, totaling approximately 10 hours of video with spatial audio. The data is categorized into SP-RACING (6.8 hours of car racing) and SP-MUSIC (2.5 hours of musical performances). Videos were identified using keywords like "binaural" and "3D audio" to ensure reliable ground-truth spatial data for training.

Performance was evaluated using two primary metrics: STFT distance (to measure spectral accuracy) and Envelope (ENV) distance (to measure how well the volume "shape" matches the ground truth).

Results

Quantitative results demonstrate that the ASN consistently outperforms both the baseline mono audio and an ablated version of the network without the classifier. In testing clips where spatial cues are most prominent, the ASN improved STFT distance by 13.4% and ENV distance by 14.8% over the mono baseline.

Quantitative comparison of the proposed ASN against baseline methods on SP-MUSIC and SP-RACING datasets.

Qualitative visualizations confirm these findings, showing that the model successfully shifts audio energy to the left or right in perfect synchronization with the visual position of racing cars or dominant musicians. The inclusion of optical flow proved critical, as it allows the model to perceive not just where an object is, but how it is moving, which is essential for dynamic spatialization.

Conclusion and Future Work

The ASN framework demonstrates that self-supervision via a correspondence classifier can effectively bridge the gap between mono and spatial audio for general videos. While the current model excels in high-motion environments like racing and music, future work will focus on more subtle "everyday" scenes and complex scenarios involving multiple overlapping sound sources moving in different directions.

Summary updated

Abstract

Spatial audio is an essential medium to audiences for 3D visual and auditory experience. However, the recording devices and techniques are expensive or inaccessible to the general public. In this work, we propose a self-supervised audio spatialization network that can generate spatial audio given the corresponding video and monaural audio. To enhance spatialization performance, we use an auxiliary classifier to classify ground-truth videos and those with audio where the left and right channels are swapped. We collect a large-scale video dataset with spatial audio to validate the proposed method. Experimental results demonstrate the effectiveness of the proposed model on the audio spatialization task.
BibTeX
@inproceedings{lu2019self,
  author = {Yu-Ding Lu and Hsin-Ying Lee and Hung-Yu Tseng and Ming-Hsuan Yang},
  title = {{Self-supervised Audio Spatialization with Correspondence Classifier}},
  booktitle = {ICIP},
  year = {2019},
  doi = {10.1109/icip.2019.8803494},
}
Figure