Improving Source Separation by Explicitly Modeling Dependencies Between Sources

Ethan Manilow, Curtis Hawthorne, Cheng-Zhi Anna Huang, Bryan Pardo, Jesse Engel

TL;DR

This paper reframes music source separation as an autoregressive problem using an Orderless Neural Autoregressive Density Estimator (NADE). Instead of isolating instruments independently, the model learns the conditional dependencies between different instruments by randomly masking sources during training and iteratively refining its predictions via block Gibbs sampling at inference time. This approach yields significant performance improvements over standard independent separation baselines.

What Problem Does This Paper Address?

In a musical piece, instruments are inherently coordinated—they share rhythms, follow harmonic structures, and frequently overlap in time and frequency. For instance, a bass guitar and a piano might play notes in the same chord, causing their higher harmonics to blend together.

Traditional deep learning models for music source separation largely ignore this context. They typically predict each instrument independently, using only the full mixture as input. This means the model cannot leverage information about one instrument to help "explain away" ambiguities in another. The authors address this gap by designing a system that explicitly models the relationships between all combinations of sources within a mixture.

Methods

The authors reframe source separation using an Orderless Neural Autoregressive Density Estimator (NADE) framework paired with Block Gibbs Sampling:

  1. Orderless NADE Training: During training, the model is given the audio mixture along with a random subset of the true "ground truth" source waveforms (called teacher forcing). The remaining sources are masked out. The network is tasked with predicting only the masked sources. By changing which sources are masked at each iteration, the network learns the conditional dependencies between all combinations of instruments.
  2. Block Gibbs Sampling (Inference): At test time, the model starts with no knowledge of individual sources (all sources masked) and generates an initial estimate. It then iteratively feeds its own previous estimates back into the network as conditioning inputs. An annealing schedule randomly masks these estimates with a probability that decreases linearly over the sampling steps, allowing the model to bootstrap and progressively refine its predictions.
Fig. 1: The proposed Orderless NADE training (top) and block Gibbs sampling procedure (bottom). For Gibbs Step N, un-masked source inputs are the model’s source estimates from Step N-1.

The authors implemented this method by modifying Demucs v2, a popular waveform-to-waveform U-Net architecture. They added 8 extra input channels: 4 for injecting the source estimates and 4 to act as "sentinel flags" indicating to the network which sources are currently masked.

Datasets

The approach was evaluated on two major source separation datasets:

  • MUSDB18: A standard benchmark consisting of 150 professional full-length tracks featuring real studio recordings of live musicians (separated into vocals, drums, bass, and other).
  • Slakh2100: A dataset containing 2,100 synthesized tracks generated from MIDI using professional-grade virtual instruments (focusing on bass, drums, guitar, and piano).

Key Insights and Ideas

  • "Explaining Away" in Audio: The core psychological/probabilistic mechanism at play is "explaining away." If the model is highly confident about the piano track, subtracting those frequencies mentally makes it significantly easier to isolate a overlapping guitar track.
  • Bootstrapping Without Architecture Changes: The model can iteratively improve its own performance purely through inference-time loops. A standard network architecture can achieve superior results simply by changing how it is trained (masking) and sampled (Gibbs loops).
  • Cross-Source Synergy: The biggest improvements happen between instruments that share the same pitch register (e.g., piano and guitar), confirming that the model successfully leverages shared contextual information to resolve frequency overlaps.

Key Results

The proposed method consistently outperformed the standard Demucs baseline after roughly 16 Gibbs steps:

  • Vocals Boost: The most dramatic improvement was observed in the MUSDB18 vocals track, which saw a massive +7.4 dB increase in Scale-Invariant Source-to-Distortion Ratio improvement (SI-SDRi) after 512 Gibbs steps.
  • Pitch Register Benefits: On the Slakh2100 dataset, piano and guitar estimates improved by over +2.0 dB.
  • Ground Truth Injection: To prove the network learned dependencies, the authors injected the actual ground truth of one instrument during inference. Doing so instantly boosted the separation quality of all other instruments (e.g., providing the true piano track improved the guitar estimate by +7.1 dB).
Fig. 2: Spectrograms showing a piano estimate after the 1st Gibbs step (left) and the 128th Gibbs step (middle). A spectrogram showing the ground truth signal is on the right. The Gibbs sampling iteratively improves the source estimation.

Limitations and Future Work

  • Inference Computation Cost: Because Gibbs sampling is sequential and cannot be parallelized, inference time scales linearly with the number of steps. The authors noted diminishing returns after 64 steps, making 512-step inference computationally expensive for real-time applications.
  • Baseline Ceiling: Instruments that the baseline model already separates exceptionally well (like drums in Slakh2100) showed little to no improvement, suggesting the method is most useful for highly ambiguous, overlapping sources.
  • Future Directions: The authors suggest exploring more advanced generative modeling frameworks (like diffusion models) to capture these multi-source dependencies more efficiently.

Summary updated

Abstract

We propose a new method for training a supervised source separation system that aims to learn the interdependent relationships between all combinations of sources in a mixture. Rather than independently estimating each source from a mix, we reframe the source separation problem as an Orderless Neural Autoregressive Density Estimator (NADE), and estimate each source from both the mix and a random subset of the other sources. We adapt a standard source separation architecture, Demucs, with additional inputs for each individual source, in addition to the input mixture. We randomly mask these input sources during training so that the network learns the conditional dependencies between the sources. By pairing this training method with a block Gibbs sampling procedure at inference time, we demonstrate that the network can iteratively improve its separation performance by conditioning a source estimate on its earlier source estimates. Experiments on two source separation datasets show that training a Demucs model with an Orderless NADE approach and using Gibbs sampling (up to 512 steps) at inference time strongly outperforms a Demucs baseline that uses a standard regression loss and direct (one step) estimation of sources.
BibTeX
@inproceedings{manilow2022improving,
  author = {Ethan Manilow and Curtis Hawthorne and Cheng-Zhi Anna Huang and Bryan Pardo and Jesse Engel},
  title = {{Improving Source Separation by Explicitly Modeling Dependencies Between Sources}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747076},
}
Figure