Abstract
Music source separation is important for applications such as karaoke and remixing. Much of previous research focuses on estimating short-time Fourier transform (STFT) magnitude and discarding phase information. We observe that, for singing voice separation, phase can make considerable improvement in separation quality. This paper proposes a complex ratio masking method for voice and accompaniment separation. The proposed method employs DenseUNet with self attention to estimate the real and imaginary components of STFT for each sound source. A simple ensemble technique is introduced to further improve separation performance. Evaluation results demonstrate that the proposed method outperforms recent state-of-the-art models for both separated voice and accompaniment.
Introduction
Singing voice separation (SVS) is the task of isolating vocals from the instrumental accompaniment in a music track. Historically, most deep learning approaches have focused solely on the magnitude spectrogram, discarding the phase information and simply reusing the noisy phase of the original mixture during reconstruction.
This research identifies two major gaps in existing methodologies:
- The Phase Gap: While it was long assumed that magnitude carries enough information for separation, empirical evidence shows that using "clean" phase instead of "mixture" phase can improve separation quality by 4–5 dB.
- The Repetition Gap: Music is highly repetitive. Standard convolutional neural networks (CNNs) have a limited receptive field and often struggle to capture these long-range repetitive patterns in the accompaniment.
Proposed Method
The proposed system is built on three pillars:
Complex Ratio Masking (cRM)
Instead of predicting a real-valued mask for the magnitude, the network estimates a complex-valued mask. This allows the model to reconstruct both the magnitude and the phase of the target source simultaneously in the complex domain. The authors found that defining the loss function in terms of the complex spectrogram error performed better than simply predicting mask values.
SA-DenseUNet Architecture
The core model is a U-Net variant that uses Dense Blocks to improve feature reuse. Crucially, it integrates Self-Attention (SA) modules. These modules allow the network to "look" at distant parts of the spectrogram to identify repetitive instrumental patterns, which helps in cleaner vocal isolation.
Multi-Context Averaging (MCA)
This ensemble technique involves training multiple models using different STFT window lengths (32ms, 64ms, and 128ms). By averaging the time-domain outputs, the system benefits from both high temporal resolution (short windows) and high frequency resolution (long windows) simultaneously, reducing artifacts and improving robustness.
Experiments and Results
The study utilized standard benchmarks including DSD100, MedleyDB, and CCMixter, augmented with 400 additional remixed songs. The proposed method significantly outperformed previous state-of-the-art models on the DSD100 test set:
- Vocal Separation: Achieved a median SDR of 9.78 dB, a substantial jump over the previous best of 7.72 dB.
- Accompaniment Separation: Achieved a median SDR of 15.20 dB, compared to the previous best of 13.90 dB.
- Phase Estimation: Using the estimated phase from the complex mask improved the SDR by nearly 0.7–0.9 dB compared to using the mixture phase.
- Temporal Resolution: Smaller frame shifts (higher overlap) during STFT consistently improved performance due to the implicit ensemble effect.
Conclusion
This paper demonstrates that phase information is critical for high-quality singing voice extraction. By employing complex ratio masking within a self-attention-enhanced DenseUNet, the authors achieved a new state-of-the-art. Future work may address the computational complexity of the ensemble approach and explore full-bandwidth complex masking at higher sampling rates.
Summary updated