Off-Screen Sound Separation Based on Audio-visual Pre-training Using Binaural Audio

Masaki Yoshida, Ren Togo, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper introduces a novel method for separating sounds originating from outside the camera's view (off-screen sounds) by leveraging audio-visual pre-training with binaural audio. By artificially generating off-screen sounds and training a model to detect their presence and direction, the researchers significantly improve the ability to isolate these sounds from on-screen audio.

What Problem Does This Paper Address?

In audio-visual analysis, a key challenge is separating and understanding sounds within a video. Traditional methods often rely on the strong correlation between what we see and what we hear – for example, seeing a person speak helps us identify their voice. However, this breaks down when sounds come from outside the camera's frame (off-screen sounds, or OSSs). These sounds lack direct visual cues, making them difficult to separate using conventional audio-visual techniques. This not only degrades the performance of audio-visual learning tasks but also can be distracting for viewers. The paper aims to solve this by developing a method that can effectively detect and separate these off-screen sounds, even when they have no visual correspondence.

Methods

The core of the proposed method lies in a two-stage approach:

  1. Audio-Visual Pre-training for Off-Screen Sound Detection:
    • Data Synthesis: The authors first address the lack of real-world data with labeled off-screen sounds. They synthesize "off-screen sounds" by taking existing on-screen audio and manipulating its arrival direction using a technique called PseudoBinaural. This allows them to create a dataset with controlled off-screen sound presence and direction.
  2. A-V Network Training: An audio-visual network is trained to detect whether an off-screen sound is present in a given audio-visual mixture. This network learns to associate visual information with the spatial characteristics of sound, specifically focusing on mismatches between visual and auditory cues that indicate an off-screen source. This pre-training step is crucial for learning a representation that captures off-screen sound characteristics.
  3. Off-Screen Sound Separation:
    • Leveraging Pre-trained Representation: The learned representation from the detection phase is then used to train a sound separation model.
  4. Mix-and-Separate Framework: The separation model employs a modified "mix-and-separate" framework. It takes the on-screen visual frames and the mixed audio (on-screen sound + synthesized OSS) as input.
  5. Binaural Audio Input: The model utilizes binaural audio, which captures interaural time and level differences, providing crucial spatial cues for sound localization.
  6. U-Net Architecture: A U-Net architecture is used to predict complex masks for both on-screen and off-screen audio components. These masks are then applied to the input spectrogram to separate the audio streams. The synthesized OSSs serve as ground truth for training this separation process.

Datasets

  • FAIR-Play Dataset: This dataset, consisting of videos with binaural audio, was used as the base for on-screen sounds. It was chosen because it contains only on-screen audio, allowing the researchers to introduce and control off-screen sounds.
  • Synthesized Dataset (Dmixture): The authors generated their own dataset by artificially adding synthesized off-screen sounds to the FAIR-Play dataset. This synthesized data was used for both the pre-training (OSS detection) and the final OSS separation tasks. This approach circumvents the difficulty of collecting real-world data with ground truth for off-screen sounds.

Key Insights and Ideas

  • The "Missing Link" of Off-Screen Sounds: The paper highlights that off-screen sounds are a significant blind spot in traditional audio-visual learning due to the lack of direct visual correspondence.
  • Binaural Audio as a Spatial Cue: The use of binaural audio is critical, as it inherently encodes spatial information (interaural time and level differences) that the model can leverage to understand sound direction.
  • Pre-training for Detection is Key: Training a model to detect off-screen sounds first is a powerful way to learn a representation that captures their unique characteristics, which then aids in their separation.
  • Artificial Data Generation is Essential: The lack of readily available datasets with ground truth for off-screen sounds necessitates creative data synthesis techniques, which the authors successfully employ.
  • Spatial Correspondence is Paramount: The success of the method underscores the importance of combining visual and auditory spatial information for robust audio-visual tasks.

Key Results

  • Improved OSS Detection: The proposed pre-training method achieved a high Area Under the ROC Curve (AUC) of 0.94 for OSS detection, outperforming methods that relied solely on mono audio or lacked visual information.
  • Effective OSS Separation: In terms of SI-SDR (a metric for separation accuracy), the proposed method (PM) achieved an average of 5.7 dB, significantly outperforming baseline audio-visual separation methods (CM1, CM2) and even some methods that used pre-trained representations but not specifically for OSS detection.
  • Directional Performance: The separation performance generally improved as the azimuth angle of the off-screen sound increased, indicating the model's ability to handle sounds from various directions.
  • Qualitative Results: Spectrogram visualizations showed that the predicted separated audio closely matched the ground truth, demonstrating the method's effectiveness in isolating off-screen sounds.

Limitations and Future Work

  • Training Data Dependency: The method currently relies on training data consisting solely of on-screen sounds, which requires either laborious labeling or capturing audio in environments without off-screen sounds.
  • Real-World Testing: While the method was evaluated on synthesized data, testing with real-world audio containing actual off-screen sounds is a crucial next step.
  • Data Generation Pipeline: The authors suggest that future work could involve building pipelines using audio and 3D environment simulators (e.g., SoundSpaces, Habitat) to generate larger and more diverse datasets for training. This would help overcome the limitations of real-world data collection.

Summary updated

Abstract

This study proposes a novel off-screen sound separation method based on audio-visual pre-training. In the field of audio-visual analysis, researchers have leveraged visual information for audio manipulation tasks, such as sound source separation. Although such audio manipulation tasks are based on correspondences between audio and video, these correspondences are not always established. Specifically, sounds coming from outside a screen have no audio-visual correspondences and thus interfere with conventional audio-visual learning. The proposed method separates such off-screen sounds based on their arrival directions using binaural audio, which provides us with three-dimensional sensation. Furthermore, we propose a new pre-training method that can consider the off-screen space and use the obtained representation to improve off-screen sound separation. Consequently, the proposed method can separate off-screen sounds irrespective of the direction from which they arrive. We conducted our evaluation using generated video data to circumvent the problem of difficulty in collecting ground truth for off-screen sounds. We confirmed the effectiveness of our methods through off-screen sound detection and separation tasks.
BibTeX
@inproceedings{yoshida2023off,
  author = {Masaki Yoshida and Ren Togo and Takahiro Ogawa and Miki Haseyama},
  title = {{Off-Screen Sound Separation Based on Audio-visual Pre-training Using Binaural Audio}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=Fmu9ISusAUf},
}
Figure