The Sound of Motions.

Hang Zhao, Chuang Gan, Wei-Chiu Ma, Antonio Torralba

Abstract

Sounds originate from object motions and vibrations of surrounding air. Inspired by the fact that humans is capable of interpreting sound sources from how objects move visually, we propose a novel system that explicitly captures such motion cues for the task of sound localization and separation. Our system is composed of an end-to-end learnable model called Deep Dense Trajectory (DDT), and a curriculum learning scheme. It exploits the inherent coherence of audio-visual signals from a large quantities of unlabeled videos. Quantitative and qualitative evaluations show that comparing to previous models that rely on visual appearance cues, our motion based system improves performance in separating musical instrument sounds. Furthermore, it separates sound components from duets of the same category of instruments, a challenging problem that has not been addressed before.

Introduction: The Cocktail Party Problem

The "cocktail party problem"—the ability to focus on one sound source amidst a cacophony—is a classic challenge in signal processing. While recent AI models have used "visual appearance" to help (e.g., identifying a guitar to isolate guitar sounds), they often fail in scenarios where sources look identical.

Separating sounds from a violin duet using motion trajectories. (a) shows the ambiguity of a single frame; (b) illustrates how motion trajectories provide the necessary cues to isolate individual waveforms.

Imagine a violin duet: both players hold the same instrument in a similar environment. In a single static frame, it is nearly impossible to determine which violin is producing which part of the audio. The key to disambiguation lies in temporal synchronization: the rhythm of the physical motion must match the rhythm of the sound. Our research introduces a system that leverages these motion cues to achieve state-of-the-art sound separation.

Methodology: Deep Dense Trajectories

We propose an end-to-end learnable framework consisting of four main components designed to prioritize motion over static appearance:

  1. Motion Network (Deep Dense Trajectory - DDT): Unlike previous models that use simple optical flow, DDT tracks pixel-wise trajectories over time. It utilizes a learnable flow network (PWC-Net) to estimate movement, tracks these points to create "dense trajectories," and then employs a 3D-CNN (I3D) to extract high-level spatiotemporal features.
  2. Appearance Network: A standard ResNet-18 architecture extracts semantic information from the first frame of the video to identify the objects present.
  3. Attention-based Fusion Module: This module uses appearance features to "gate" the motion features, directing the model to focus on relevant trajectories (e.g., moving hands) rather than background noise.
  4. Sound Separation Network: A U-Net-based architecture takes a mixed audio spectrogram and uses the fused visual features to predict a binary mask, which is then applied to isolate the target sound.

Self-Supervised Curriculum Learning

The model is trained via a "Mix-and-Separate" strategy using unlabeled video. To handle the most difficult cases, we implemented a 3-stage curriculum learning scheme. The model begins by learning to separate vastly different instruments (e.g., a flute from a drum) before progressing to the more nuanced task of separating different performances of the same instrument type.

Key Results and Insights

Our system was evaluated on the MUSIC-21 and URMP datasets, yielding significant improvements over appearance-based baselines:

  • State-of-the-Art Performance: On general instrument separation, the DDT model achieved a Signal-to-Distortion Ratio (SDR) of 8.31 dB, surpassing the previous state-of-the-art "Sound of Pixels" (7.52 dB).
  • The "Same Instrument" Breakthrough: In the most challenging scenario—separating duets of the same instrument—the model showed a massive improvement. For violin duets, it achieved an SDR of 6.33 dB compared to just 1.95 dB for appearance-only models.
  • Motion as a Discriminator: While appearance identifies what an object is, motion identifies when it is active. This temporal alignment allows the model to correctly ignore "silent" instruments (like a guitar hanging on a wall) that often confuse other systems.

Conclusion and Future Work

By explicitly modeling motion trajectories, we have addressed a significant limitation in audio-visual sound separation. While highly effective for rhythmic musical performances, future work will explore more stochastic sounds, such as footsteps or environmental noises, and address challenges related to off-screen or occluded sound sources.

Summary updated

Abstract

Sounds originate from object motions and vibrations of surrounding air. Inspired by the fact that humans is capable of interpreting sound sources from how objects move visually, we propose a novel system that explicitly captures such motion cues for the task of sound localization and separation. Our system is composed of an end-to-end learnable model called Deep Dense Trajectory (DDT), and a curriculum learning scheme. It exploits the inherent coherence of audio-visual signals from a large quantities of unlabeled videos. Quantitative and qualitative evaluations show that comparing to previous models that rely on visual appearance cues, our motion based system improves performance in separating musical instrument sounds. Furthermore, it separates sound components from duets of the same category of instruments, a challenging problem that has not been addressed before.
BibTeX
@inproceedings{zhao2019sound,
  author = {Hang Zhao and Chuang Gan and Wei-Chiu Ma and Antonio Torralba},
  title = {{The Sound of Motions.}},
  booktitle = {ICCV},
  year = {2019},
  url = {https://openreview.net/forum?id=U4_M4I6gZh9},
}
Figure