Music Source Separation With Deep Equilibrium Models

Yuichiro Koyama, Naoki Murata, Stefan Uhlich, Giorgio Fabbro, Shusuke Takahashi, Yuki Mitsufuji

TL;DR

Researchers from Sony have successfully applied Deep Equilibrium Models (DEQ) to music source separation, creating a model called DEQ-UMX. By replacing the standard multi-layer stack in the popular Open-Unmix (UMX) architecture with an "implicit" layer that solves for an equilibrium point, they achieved a 30% reduction in parameters while actually improving separation performance (SDR).

What Problem Does This Paper Address?

Music Source Separation (MSS)—the task of isolating vocals, drums, bass, and other instruments from a mixed track—is dominated by large Deep Neural Networks (DNNs). While effective, these models are often "heavy," making them difficult to deploy on mobile devices or distribute in software packages due to high memory usage and large file sizes.

The challenge is to reduce the model size (parameter count) without sacrificing the quality of the separation. Standard "explicit" networks stack many layers to gain expressive power, but each layer adds more parameters. The authors look toward Deep Equilibrium Models, a class of "implicit" architectures that can simulate the behavior of an infinitely deep network using the parameters of just a single layer.

Methods

The authors base their work on Open-Unmix (UMX), a well-known baseline in the field. They specifically target the "sequence modeling" block, which usually consists of three stacked Bidirectional LSTM (BLSTM) layers.

The DEQ Approach

Instead of stacking layers, a DEQ defines its output as the equilibrium point (fixed point) of a single transformation function $f_\theta$. Mathematically, it finds a state where $z = f_\theta(z; x)$.

  • Forward Pass: Instead of passing data through Layer 1 $\rightarrow$ Layer 2 $\rightarrow$ Layer 3, the model uses a root-finding algorithm (like Broyden’s method) to iteratively find the stable state where the output no longer changes.
  • Core Function ($f_\theta$): The authors designed a custom core function for audio, using a single-layer BLSTM combined with Group Normalization (GN) and a concatenation block to merge the input sequence with the hidden state.
  • Backward Pass: To train the model efficiently, they used Jacobian-Free Backpropagation (JFB). This avoids the massive memory cost of storing intermediate states and is more robust to estimation errors during the root-finding process.

Datasets

The study uses the industry-standard MUSDB18 dataset.

  • Content: 150 professionally recorded full-length songs.
  • Stems: Each song includes isolated tracks for vocals, drums, bass, and "other."
  • Setup: 86 songs for training, 14 for validation, and 50 for the test set. All audio was processed at 44.1 kHz.

Key Insights and Ideas

  • BLSTM as a Shortcut to Equilibrium: Usually, DEQs (like those using Convolutions) require many iterations to reach a stable state. However, because a BLSTM processes the entire sequence at once, it captures global context very quickly. The authors found that DEQ-UMX reaches peak performance with very few iterations (low "Number of Function Evaluations" or NFE).
  • Normalization Matters: They swapped Batch Normalization (BN) for Group Normalization (GN). In DEQs, BN can lead to "poor conditioning" of the underlying math (the Jacobian matrix), making it harder for the model to converge. GN proved much more stable.
  • Parameter Efficiency vs. Compute: The "aha moment" is that weight-tying (using the same parameters repeatedly) isn't just a compression trick; when framed as an equilibrium problem, it can actually find a "better" mathematical representation of the audio than a standard stack of different layers.

Key Results

The proposed DEQ-UMX was compared against the original UMX and several variants:

  • Better Performance: DEQ-UMX achieved an average Signal-to-Distortion Ratio (SDR) of 5.53 dB, outperforming the original UMX (5.33 dB).
  • Smaller Footprint: It reduced the number of parameters from 35.55M to 25.06M (a ~30% reduction).
  • Robustness: Even compared to a "UMX small" (which has a similar parameter count), DEQ-UMX performed significantly better, proving that the implicit architecture is more "expressive" than simply shrinking a standard network.
Method Parameters (M) Avg. SDR (dB)
Open-Unmix (UMX) 35.55 5.33
UMX small 25.15 5.24
DEQ-UMX (Proposed) 25.06 5.53

Limitations and Future Work

  • Computational Complexity: While the model has fewer parameters (saving disk space and RAM), it actually requires more MACs (Multiply-Accumulate operations) because of the iterative root-finding. This means it might be slower to run on a CPU/GPU than the original UMX.
  • Root-Finding Efficiency: The authors acknowledge that finding more efficient root-finding schemes is a priority to reduce the "computational overhead" during inference.
  • Generalization: The current study focuses on the UMX architecture. Future work involves applying the DEQ framework to more modern, state-of-the-art architectures to see if the gains in parameter efficiency hold up across different designs.

Summary updated

Abstract

While deep neural network-based music source separation (MSS) is very effective and achieves high performance, its model size is often a problem for practical deployment. Deep implicit architectures such as deep equilibrium models (DEQ) were recently proposed, which can achieve higher performance than their explicit counterparts with limited depth while keeping the number of parameters small. This makes DEQ also attractive for MSS, especially as it was originally applied to sequential modeling tasks in natural language processing and thus should in principle be also suited for MSS. However, an investigation of a good architecture and training scheme for MSS with DEQ is needed as the characteristics of acoustic signals are different from those of natural language data. Hence, in this paper we propose an architecture and training scheme for MSS with DEQ. Starting with the architecture of Open-Unmix (UMX), we replace its sequence model with DEQ. We refer to our proposed method as DEQ-based UMX (DEQ-UMX). Experimental results show that DEQ-UMX performs better than the original UMX while reducing its number of parameters by 30%.
BibTeX
@inproceedings{koyama2022music,
  author = {Yuichiro Koyama and Naoki Murata and Stefan Uhlich and Giorgio Fabbro and Shusuke Takahashi and Yuki Mitsufuji},
  title = {{Music Source Separation With Deep Equilibrium Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746317},
}
Figure