TL;DR
Researchers have introduced MedleyVox, the first dedicated evaluation benchmark for separating multiple overlapping singing voices (e.g., duets or unisons). Alongside the dataset, they propose iSRNet, a lightweight super-resolution network that significantly improves separation quality, and a novel strategy for synthesizing training data from single-voice recordings.
What Problem Does This Paper Address?
While AI has become remarkably good at separating a single lead vocal from a musical accompaniment, separating multiple singing voices from each other remains a "rarely studied" frontier.
Unlike speech separation (where speakers usually have different pitches and rhythms), singing voices are often highly correlated. In a duet or choir, singers might hit the exact same notes at the exact same time (unison) or follow tight harmonic structures. This makes it incredibly difficult for standard models to distinguish between sources. The field has been held back by two major hurdles:
- Lack of Data: There were no high-quality, publicly available benchmark datasets for multi-singing separation.
- Model Limitations: Existing speech separation models struggle with the wide frequency ranges and harmonic richness of singing, often producing "crunchy" artifacts or failing to handle the phase-alignment issues of unison singing.
Methods
The authors tackle the problem through a three-pronged approach:
1. The iSRNet Architecture
They propose the Improved Super-Resolution Network (iSRNet). Instead of trying to separate the audio from scratch, iSRNet acts as a "refiner." It takes the initial (often blurry or leaky) estimates from a primary network like Conv-TasNet and enhances them.
- What’s new: They replaced standard convolutional layers with ConvNeXt blocks for better efficiency.
- Heuristic Guidance: The model uses a "heuristic magnitude STFT" to help the network understand frequency boundaries, specifically targeting the 1–5kHz range where leakage is most common.
2. Training Mixture Construction
Since no large-scale multi-singing training set exists, the authors developed a strategy to synthesize one. They took over 400 hours of single-singer data and mixed them using:
- Pitch and Formant Shifting: To simulate different vocal characteristics.
- Dynamic Mixing: Randomly sampling sources to create "unison," "duet," and "main vs. rest" scenarios.
3. Optimization Tweaks
- STFT vs. Learnable Bases: They found that standard STFT/iSTFT analysis works better for singing than the "learnable" encoders popular in speech AI, which tend to produce audible artifacts.
- Mixture Consistency: They implemented a projection layer to ensure the separated outputs, when added back together, exactly equal the original mixture. This prevents "scale exploding" (where the volume of the output becomes massive/clipped).
Datasets
- MedleyVox (New): An evaluation dataset derived from MedleyDB. It contains 381 manually categorized segments (nearly 4,000 seconds) divided into:
- Unison: Two voices with identical or octave-shifted melodies.
- Duet: Two voices with different melodies/lyrics.
- Main vs. Rest: One lead singer vs. a group of background singers.
- N-singing: An unknown number of singers.
- Training Data: A massive compilation of 400 hours from 13 single-singing datasets (including MedleyDB, VocalSet, and various Japanese/Korean singing databases) plus 460 hours of LibriSpeech.
Key Insights and Ideas
- Unison is the "Final Boss": The results show that separating unison voices is significantly harder than duets. SDR (Signal-to-Distortion Ratio) scores for unison were roughly 10 dB lower than for duets. This highlights that phase information is the critical missing link for unison separation.
- Efficiency Matters: By using ConvNeXt blocks, iSRNet achieved better performance than previous super-resolution models while using 40x fewer parameters and significantly less computational power (47.5G MACs vs 1.93T).
- Singing $\neq$ Speech: Models optimized for speech often fail on singing because singers hold notes longer and have much wider fundamental frequency ranges. The authors found that "waveform-based" models (standard in speech) are prone to "crunchy noise" when applied to the harmonic complexity of song.
Key Results
- Duet Separation: The iSRNet model achieved a 15.1 dB SDR improvement (SDRi), outperforming the baseline Conv-TasNet (14.2 dB).
- Unison Separation: While harder, the model still achieved a 4.9 dB SDRi, which actually outperformed "ideal" ratio masks (IRM), suggesting the model is learning to handle complex phase relationships better than simple frequency filters.
- Main vs. Rest: The model achieved a 7.2 dB SDRi. While helpful for applications like karaoke (removing a lead vocal while keeping backing vocals), it still lags behind "oracle" methods, leaving room for improvement.
Limitations and Future Work
- Singer Swapping: In some cases, the model "swaps" identities mid-song (e.g., assigning Singer A’s voice to Output B halfway through). The authors suggest that future models need better singer representations (embeddings) to maintain identity over time.
- Silence Handling: In "Main vs. Rest" tasks, the model sometimes tries to "fill in" silent parts of the lead vocal track with leaked audio from the background singers.
- Long-form Audio: The current study focuses on short segments. Processing full 4-minute songs remains a challenge due to the memory requirements of high-resolution audio and the difficulty of maintaining consistent separation over long durations.
Summary updated