The Sound of Motions

Hang Zhao, Chuang Gan, Wei-Chiu Ma, Antonio Torralba

Abstract

Sounds originate from object motions and vibrations of surrounding air. Inspired by the fact that humans are capable of interpreting sound sources from how objects move visually, we propose a novel system that explicitly captures such motion cues for the task of sound localization and separation. Our system is composed of an end-to-end learnable model called Deep Dense Trajectory (DDT), and a curriculum learning scheme. It exploits the inherent coherence of audio-visual signals from large quantities of unlabeled videos. Quantitative and qualitative evaluations show that compared to previous models that rely on visual appearance cues, our motion-based system improves performance in separating musical instrument sounds. Furthermore, it separates sound components from duets of the same category of instruments, a challenging problem that has not been addressed before.

Introduction: Beyond Visual Appearance

The "Cocktail Party Problem"—the ability to focus on a single auditory source amidst a cacophony of others—is a classic challenge in signal processing. While recent deep learning models have made strides in audio-visual sound separation, they mostly rely on visual appearance. For instance, a model might learn that "pixels looking like a guitar" correlate with "guitar sounds."

However, appearance-based models fail in scenarios where the sources look nearly identical. If two people play violins in the same frame, their visual semantics are identical. To distinguish them, a human listener uses motion: we associate the specific rhythm of a bow stroke with the specific melody we hear. This research addresses the gap in current AI by explicitly modeling these temporal motion cues to disambiguate and separate sounds.

Separating sounds from a violin duet using visual motion trajectories.

Methodology

The proposed framework consists of three main pillars designed to capture and utilize motion for sound separation.

1. Deep Dense Trajectory (DDT) Network

Instead of analyzing raw pixels, this network extracts motion. It utilizes a pre-trained optical flow model (PWC-Net) to track the movement of every pixel across frames. These paths, or trajectories, are then processed by an Inflated 3D ConvNet (I3D) to capture spatiotemporal features.

2. Mix-and-Separate Framework

This self-supervised training strategy involves taking audio from two different video clips, mixing them, and then training the system to recover the original audio of one clip using only that clip's visual motion as a guide.

3. Curriculum Learning

Separating identical instruments is inherently difficult. The authors employ a three-stage training process:

  • Stage 1: Separate sounds from different instrument categories (e.g., tuba vs. flute).
  • Stage 2: Separate sounds from the same category but different videos (e.g., two different cellos).
  • Stage 3: Separate sounds from the same video (e.g., a duet), where motion is the primary distinguishing factor.

The final separation is performed by a U-Net that predicts a binary spectrogram mask, conditioned on visual features via Feature-wise Linear Modulation (FiLM) layers.

Overview of the proposed system architecture, including Motion, Appearance, and Sound Separation networks.

Experimental Setup and Datasets

The study utilizes two primary datasets:

  • MUSIC and MUSIC-21: A collection of YouTube videos featuring 21 categories of musical solos and duets.
  • URMP (University of Rochester Multi-Modal Music Performance): A high-quality dataset of multi-instrument ensemble performances recorded in a studio setting with individual tracks for each instrument, providing a "gold standard" for evaluation.

Key Results

The "Sound of Motions" model significantly outperformed previous state-of-the-art methods, such as Sound of Pixels:

  • Quantitative Superiority: In a 2-source mixture, the model achieved a Signal-to-Distortion Ratio (SDR) of 8.31 dB, compared to 7.52 dB for appearance-only models.
  • The Duet Challenge: In the difficult task of separating duets of the same instrument, the model achieved an SDR of 6.33 dB for violins, whereas previous models struggled at 1.95 dB.
  • Human Evaluation: In Amazon Mechanical Turk tests, human listeners preferred the separation results of this motion-based model over previous methods by a wide margin (e.g., 61.25% vs 38.75% for violins).

Discussion and Limitations

A byproduct of the model is its ability to localize sound. Heatmaps show the model "listening" specifically to the parts of the instrument being interacted with (like violin strings) while ignoring silent background instruments.

However, limitations remain. The model focuses on interaction points rather than the "whole" object (e.g., the vibrating string vs. the entire violin body). Additionally, calculating dense trajectories for every pixel is computationally expensive, posing a challenge for real-time applications. Future work could explore more chaotic, "in-the-wild" environments where motion cues might be noisier.

Conclusion

By moving beyond static image recognition and focusing on how objects move over time, the "Sound of Motions" system successfully performs the challenging task of separating sounds from identical instruments playing simultaneously. This highlights the critical role of temporal motion cues in solving the cocktail party problem.

Summary updated

Abstract

Sounds originate from object motions and vibrations of surrounding air. Inspired by the fact that humans is capable of interpreting sound sources from how objects move visually, we propose a novel system that explicitly captures such motion cues for the task of sound localization and separation. Our system is composed of an end-to-end learnable model called Deep Dense Trajectory (DDT), and a curriculum learning scheme. It exploits the inherent coherence of audio-visual signals from a large quantities of unlabeled videos. Quantitative and qualitative evaluations show that comparing to previous models that rely on visual appearance cues, our motion based system improves performance in separating musical instrument sounds. Furthermore, it separates sound components from duets of the same category of instruments, a challenging problem that has not been addressed before.
BibTeX
@inproceedings{zhao2019sound,
  author = {Hang Zhao and Chuang Gan and Wei-Chiu Ma and Antonio Torralba},
  title = {{The Sound of Motions}},
  booktitle = {ICCV},
  year = {2019},
  url = {https://openreview.net/forum?id=U4_M4I6gZh9},
}
Figure