Complex Ratio Masking For Singing Voice Separation

Yixuan Zhang, Yuzhou Liu, DeLiang Wang

TL;DR

This paper demonstrates that phase information—often discarded in music source separation—is critical for high-quality singing voice extraction. By employing a Complex Ideal Ratio Mask (cIRM) within a self-attention-enhanced DenseUNet and using an ensemble of models with varying window lengths, the authors achieved a new state-of-the-art Signal-to-Distortion Ratio (SDR) of 9.78 dB on the DSD100 dataset.

What Problem Does This Paper Address?

Singing voice separation (SVS) is the task of isolating vocals from the instrumental accompaniment in a music track. Historically, most deep learning approaches have focused solely on the magnitude spectrogram, discarding the phase information and simply reusing the noisy phase of the original mixture during reconstruction.

The authors identify two major gaps:

  1. The Phase Gap: While it was long assumed that magnitude carries enough information for separation, the authors show that using "clean" phase instead of "mixture" phase can improve separation quality by 4–5 dB.
  2. The Repetition Gap: Music, unlike speech, is highly repetitive. Standard convolutional neural networks (CNNs) have a limited receptive field and often struggle to capture these long-range repetitive patterns in the accompaniment.

Methods

The authors propose a system built on three pillars:

  1. Complex Ratio Masking (cRM): Instead of predicting a real-valued mask for the magnitude, the network estimates a complex-valued mask. This allows the model to reconstruct both the magnitude and the phase of the target source simultaneously in the complex domain.
  2. SA-DenseUNet Architecture: The core model is a U-Net variant that uses Dense Blocks (where each layer is connected to every other layer in the block) to improve feature reuse. Crucially, they integrate Self-Attention (SA) modules. These modules allow the network to "look" at distant parts of the spectrogram to identify repetitive instrumental patterns, which helps in cleaner vocal isolation.
  3. Multi-Context Averaging (MCA): This is an ensemble technique. The authors trained multiple models using different Short-Time Fourier Transform (STFT) window lengths (32ms, 64ms, and 128ms). By averaging the time-domain outputs of these models, they reduce artifacts and improve overall robustness.

Datasets

The study utilized several standard benchmarks in the music information retrieval (MIR) community:

  • DSD100: A dataset of 100 full-track songs (split into Dev and Test sets) used for the Signal Separation Evaluation Campaign (SiSEC).
  • MedleyDB: A multitrack dataset used here to augment the training data.
  • CCMixter: Another common dataset for vocal separation.
  • Data Augmentation: The authors generated 400 additional songs by randomly scaling and remixing sources from these datasets to prevent overfitting.

Key Insights and Ideas

  • Phase Matters More Than We Thought: The paper provides empirical evidence (Figure 1) that the "magnitude-only" assumption is a significant bottleneck. Moving to the complex domain provides a direct path to higher SDR.
  • Attention is for Accompaniment: While CNNs are great at local textures (like the grain of a voice), self-attention is uniquely suited for music because it can link a drum beat or guitar riff at the beginning of a song to its repetition minutes later.
  • Loss Function Nuance: The authors found that defining the loss function in terms of the complex spectrogram error (the difference between the target and the masked mixture) performed better than simply trying to predict the mask values themselves.
  • Temporal Resolution Trade-offs: By using Multi-Context Averaging, the model benefits from both high temporal resolution (short windows) and high frequency resolution (long windows) simultaneously.

Key Results

The proposed method significantly outperformed previous state-of-the-art models on the DSD100 test set:

  • Vocal Separation: Achieved a median SDR of 9.78 dB, a substantial jump over the previous best of 7.72 dB (SA-DenseUNet magnitude-based).
  • Accompaniment Separation: Achieved a median SDR of 15.20 dB, compared to the previous best of 13.90 dB.
  • Phase Estimation: Using the estimated phase from the complex mask improved the SDR by nearly 0.7–0.9 dB compared to just using the original mixture phase.
  • Frame Shift: They observed that smaller frame shifts (higher overlap) during STFT consistently improved performance, likely due to the implicit "ensemble" effect of averaging multiple estimates for each time sample.

Limitations and Future Work

While the results are impressive, a few areas remain for exploration:

  • Computational Complexity: The use of an ensemble (MCA) and self-attention layers increases the computational footprint, which might make real-time implementation challenging on consumer hardware.
  • Sampling Rate: Most training was done at 16 kHz to save on computation, though evaluation was performed at 44.1 kHz. Exploring full-bandwidth complex masking could yield even higher fidelity.
  • Generalization: While it works well on singing voices, it remains to be seen how this specific complex-masking architecture handles more "vocal-like" instruments (e.g., cello or saxophone) that might confuse the self-attention mechanism.

Summary updated

Abstract

Music source separation is important for applications such as karaoke and remixing. Much of previous research focuses on estimating short-time Fourier transform (STFT) magnitude and discarding phase information. We observe that, for singing voice separation, phase can make considerable improvement in separation quality. This paper proposes a complex ratio masking method for voice and accompaniment separation. The proposed method employs DenseUNet with self attention to estimate the real and imaginary components of STFT for each sound source. A simple ensemble technique is introduced to further improve separation performance. Evaluation results demonstrate that the proposed method outperforms recent state-of-the-art models for both separated voice and accompaniment.
BibTeX
@inproceedings{zhang2021complex,
  author = {Yixuan Zhang and Yuzhou Liu and DeLiang Wang},
  title = {{Complex Ratio Masking For Singing Voice Separation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=2lgPhKxTpi},
}

Loading related content...

Figure