TL;DR
This paper introduces AMnet, a novel self-supervised framework for separating and localizing sound sources in videos. By learning a joint audio-motion representation, AMnet achieves state-of-the-art performance without relying on pre-trained detectors or specific assumptions about the sound source.
What Problem Does This Paper Address?
The task of sound source separation, which aims to isolate a specific sound from a mixture, is notoriously challenging. While traditional audio-only methods struggle, recent research has explored "visually guided sound source separation," leveraging visual cues from videos to aid the process. However, existing visual methods often rely on prior knowledge about the sound source (e.g., human actions) or require pre-trained models for tasks like keypoint detection or optical flow estimation, limiting their generalizability. This paper addresses the need for a more robust and generalizable self-supervised approach that can effectively utilize visual motion information for sound separation and localization without these constraints.
Methods
The core of the proposed solution is the Appearance and Motion network (AMnet), a two-stage architecture designed to leverage both visual appearance and motion cues.
- Audio-Motion Embedding (AME) Framework: This is a key novelty. It learns a shared embedding space for audio and motion by training a "Motion Network" and a "Sound Network." The training objective uses a triplet loss to enforce that synchronized audio-motion pairs have similar embeddings, while unsynchronized pairs have dissimilar ones. This self-supervised approach encourages the motion network to focus on sound-relevant motion features without explicit supervision.
- Audio-Appearance Stage: This initial stage performs sound separation using pure appearance-based features. It encodes a video frame and a mixture audio spectrogram, fuses them, and generates a binary mask to extract the sound.
- Audio-Motion Stage: This stage refines the separation using motion cues. It employs a Motion Network (part of the AME) to extract motion representations from the video sequence. These motion features are then fused with the spectrogram features from the previous stage using a novel Audio-Motion Transformer (AMT) module. The AMT leverages multi-head attention to effectively combine audio and motion information.
- Sound Source Localization: The AME framework also enables sound source localization by learning discriminative motion features correlated with sound.
Datasets
The paper uses the following datasets for training and evaluation:
- MUSIC-21: Contains videos of 21 instrumental categories.
- Audio-Visual Event (AVE): A subset of AudioSet, featuring 4143 videos covering 28 event categories, including human activities and vehicle sounds.
The authors note that they did not introduce new benchmarks but evaluated on existing challenging datasets.
Key Insights and Ideas
- Motion is Crucial, Even Without Explicit Supervision: The AME framework demonstrates that learning a joint audio-motion embedding space can effectively capture sound-relevant motion cues without needing pre-trained motion detectors or explicit labels for motion types.
- Two-Stage Approach for Specialization: Separating the tasks into an "Audio-Appearance" stage and an "Audio-Motion" stage allows each to specialize in its respective cue, leading to better overall performance than a single-stage fusion.
- Transformer for Audio-Motion Fusion: The Audio-Motion Transformer (AMT) proves effective in integrating the learned motion representations with audio features, outperforming simpler fusion methods.
- Generalizability Beyond Human Motion: Unlike some prior work that focused on human body or finger movements, AMnet's self-supervised motion learning is general enough to capture motions from various sources, including vehicles.
Key Results
- State-of-the-Art Performance: AMnet achieves state-of-the-art results on both the MUSIC-21 and AVE datasets for visually guided sound source separation.
- Improved Localization: The AME framework shows strong performance in sound source localization, outperforming baselines like COF and Multisensory by accurately identifying motion regions correlated with sound.
- Competitive with Specialized Methods: AMnet surpasses even methods that utilize pre-trained keypoint detectors for human motion, highlighting the power of its self-supervised approach. For example, on MUSIC-21 with two sources, AMnet achieved an SDR of 11.08 dB, compared to 10.12 dB for Music Gesture [15].
- Robustness to Same-Type Sources: AMnet demonstrates superior performance in separating sounds from sources of the same type (e.g., two cellos), a challenging scenario where appearance cues are less informative.
Limitations and Future Work
- Limitations: The paper acknowledges that while the AME framework is general, the performance can still be affected by the complexity of the audio-visual scene and the number of sources. The "Copy-Paste" baseline outperforming some methods in the same-type source separation task on MUSIC-21 suggests that even AMnet has room for improvement in extremely challenging scenarios.
- Future Work: The authors suggest that the learned motion representations could be further explored for other audio-visual tasks. The current work focuses on two-stage separation; extending this to more complex multi-source scenarios or exploring different fusion mechanisms could be future directions. The paper also mentions that the AME framework can be used for action recognition, indicating potential for broader applications.
Summary updated