Improving Source Separation by Explicitly Modeling Dependencies between Sources

Ethan Manilow, Curtis Hawthorne, Cheng-Zhi Anna Huang, Bryan Pardo, Jesse H. Engel

TL;DR

This paper introduces a novel training and inference method for music source separation that explicitly models the interdependencies between different instruments. By treating source separation as an "Orderless Neural Autoregressive Density Estimator" (NADE) and employing a block Gibbs sampling strategy during inference, the system iteratively refines source estimates, leading to significantly improved separation quality over standard independent source modeling.

What Problem Does This Paper Address?

Music is inherently a coordinated art form, with instruments playing together in time and harmony. This coordination, while essential for musicality, makes the task of separating individual instruments from a mixed audio signal (music source separation) challenging. Traditional deep learning approaches often treat each source independently, essentially trying to "explain away" each instrument without leveraging the contextual information provided by others. This paper addresses the gap in current source separation methods by proposing a way to explicitly model and exploit these interdependencies, aiming to improve separation performance by allowing sources to "explain away" each other.

Methods

The core innovation lies in two key components:

  • Orderless NADE Training: Instead of independently predicting each source from the mixture, the authors reframe the problem as an Orderless Neural Autoregressive Density Estimator (NADE). During training, the network receives the mixture and a random subset of the ground truth sources as input. It is then tasked with predicting the remaining masked sources. This forces the network to learn the conditional dependencies between sources to accurately predict the masked ones. This is achieved by adapting the Demucs architecture with additional input channels for conditioning.
  • Block Gibbs Sampling Inference: At inference time, a block Gibbs sampling procedure is used. This iterative process starts with an initial set of source estimates and then repeatedly feeds these estimates back into the network, conditioning the prediction of each source on the current estimates of the others. This allows the system to refine its estimates over multiple steps, progressively improving the separation quality by leveraging the contextual information learned during training. An annealing schedule is used to gradually reduce the probability of masking sources during sampling.

Datasets

The authors evaluate their method on two publicly available datasets:

  • MUSDB18: A dataset of 150 mixtures derived from real recording sessions, featuring live musicians.
  • Slakh2100: A larger dataset of 2,100 mixtures synthesized using professional-grade sample-based synthesis engines.

Key Insights and Ideas

  • Interdependencies Matter: The paper strongly suggests that explicitly modeling the relationships between musical sources, rather than treating them in isolation, leads to significant performance gains in source separation.
  • "Explaining Away" in Audio: The concept of "explaining away" from probability theory, where the presence of one explanation reduces the likelihood of another, is shown to be applicable and beneficial in the context of audio source separation.
  • Iterative Refinement Pays Off: The block Gibbs sampling approach demonstrates that iteratively refining source estimates, even after initial separation, can lead to substantial improvements, especially for sources that are harmonically or rhythmically intertwined.
  • Orderless NADE as a Powerful Training Paradigm: The NADE training strategy, by forcing the model to predict masked sources based on others, effectively teaches it to understand contextual relationships within a musical mixture.

Key Results

The proposed method consistently outperforms a standard Demucs baseline that uses a direct regression loss and single-step estimation.

  • Significant Improvements: For most sources across both datasets, the proposed method shows substantial improvements in SI-SDRi (Scale-Invariant Source-to-Distortion Ratio) over the baseline.
  • Iterative Gains: The performance generally increases with the number of Gibbs sampling steps, although diminishing returns are observed after around 64 steps.
  • Early Wins: Even after just one Gibbs step (effectively using the NADE-trained model for a single refinement), many sources show improved separation, indicating the benefit of the NADE training itself.
  • Notable Source Improvements: The "vocals" source in MUSDB18 saw a remarkable +7.4 dB improvement after 512 Gibbs steps. In Slakh2100, "piano" and "guitar" sources improved by over +2.0 dB after 256 steps, likely due to their tendency to occupy similar frequency ranges.
  • Injecting Ground Truth: An experiment where ground truth data was injected for one source at the first Gibbs step showed that the model could effectively leverage this information, leading to improved estimates for other sources, particularly for instruments in similar pitch registers.

Limitations and Future Work

  • Inference Speed: The block Gibbs sampling procedure is not parallelized, meaning inference time scales linearly with the number of sampling steps. This limits the practical application of a very large number of steps.
  • Diminishing Returns: While performance generally improves with more Gibbs steps, the gains become smaller after a certain point, suggesting an optimal number of steps to balance performance and inference time.
  • Future Work: The authors express excitement about exploring other generative modeling ideas to further advance understanding of musical scenes. They also note that their approach is architecture-agnostic, suggesting it could be applied to other source separation models.

Summary updated

BibTeX
@inproceedings{manilow2022improving,
  author = {Ethan Manilow and Curtis Hawthorne and Cheng-Zhi Anna Huang and Bryan Pardo and Jesse H. Engel},
  title = {{Improving Source Separation by Explicitly Modeling Dependencies between Sources}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747076},
}
Figure