MedleyVox: An Evaluation Dataset for Multiple Singing Voices Separation

Chang-Bin Jeon, Hyeongi Moon, Keunwoo Choi, Ben Sangbae Chon, Kyogu Lee

Abstract: Separation of multiple singing voices into each voice is a rarely studied area in music source separation research. The absence of a benchmark dataset has hindered its progress. In this paper, we present an evaluation dataset and provide baseline studies for multiple singing voices separation. First, we introduce MedleyVox, an evaluation dataset for multiple singing voices separation. We specify the problem definition in this dataset by categorizing it into i) unison, ii) duet, iii) main vs. rest, and iv) N-singing separation. Second, to overcome the absence of existing multi-singing datasets for a training purpose, we present a strategy for construction of multiple singing mixtures using various single-singing datasets. Third, we propose the improved super-resolution network (iSRNet), which greatly enhances initial estimates of separation networks. Jointly trained with the Conv-TasNet and the multi-singing mixture construction strategy, the proposed iSRNet achieved comparable performance to ideal time-frequency masks on duet and unison subsets of MedleyVox.

Introduction

While AI has become remarkably good at separating a single lead vocal from a musical accompaniment, separating multiple singing voices from each other remains a frontier in music source separation. Unlike speech separation, where speakers usually have different pitches and rhythms, singing voices are often highly correlated. In a duet or choir, singers might hit the exact same notes at the exact same time (unison) or follow tight harmonic structures, making it incredibly difficult for standard models to distinguish between sources.

This field has been held back by two major hurdles: a lack of high-quality, publicly available benchmark datasets for multi-singing separation, and model limitations where existing speech separation models struggle with the wide frequency ranges and harmonic richness of singing.

Proposed Method: iSRNet and Training Strategy

To address these challenges, the researchers propose the Improved Super-Resolution Network (iSRNet). Rather than separating audio from scratch, iSRNet acts as a refiner that enhances initial, often leaky, estimates from a primary network like Conv-TasNet.

Architecture of the proposed iSRNet and the heuristic magnitude STFT process.

The iSRNet architecture utilizes ConvNeXt blocks for improved efficiency and incorporates a "heuristic magnitude STFT" to guide the network in understanding frequency boundaries, specifically targeting the 1–5kHz range where leakage is most common.

Because no large-scale multi-singing training set exists, the authors developed a strategy to synthesize one from over 400 hours of single-singer data. This involves pitch and formant shifting to simulate different vocal characteristics and dynamic mixing to create scenarios like unison, duet, and "main vs. rest."

The MedleyVox Benchmark

The primary contribution of this work is MedleyVox, an evaluation dataset derived from MedleyDB. It contains 381 manually categorized segments (nearly 4,000 seconds) divided into specific categories based on the relationship between the voices.

Statistics of the MedleyVox evaluation dataset across different categories.

The dataset is categorized into:

  • Unison: Two voices with identical or octave-shifted melodies.
  • Duet: Two voices with different melodies/lyrics.
  • Main vs. Rest: One lead singer vs. a group of background singers.
  • N-singing: An unknown number of singers.

Key Results and Insights

Experimental results demonstrate that iSRNet significantly improves separation quality:

  • Duet Separation: The model achieved a 15.1 dB SDR improvement (SDRi), outperforming the baseline Conv-TasNet (14.2 dB).
  • Unison Separation: This remains the most difficult task, with SDR scores roughly 10 dB lower than for duets. However, the model still achieved a 4.9 dB SDRi, outperforming ideal ratio masks (IRM), suggesting it learns complex phase relationships better than simple filters.
  • Efficiency: By using ConvNeXt blocks, iSRNet achieved superior performance while using 40x fewer parameters than previous super-resolution models.

Discussion and Future Work

While iSRNet shows promise, limitations remain. The model occasionally "swaps" singer identities mid-song, suggesting a need for better singer embeddings. Additionally, in "Main vs. Rest" tasks, the model sometimes fills silent parts of the lead vocal track with leaked audio from background singers. Future work will focus on maintaining identity consistency over long-form audio and improving the handling of silent intervals in multi-voice mixtures.

Summary updated

Abstract

Separation of multiple singing voices into each voice is a rarely studied area in music source separation research. The absence of a benchmark dataset has hindered its progress. In this paper, we present an evaluation dataset and provide baseline studies for multiple singing voices separation. First, we introduce MedleyVox, an evaluation dataset for multiple singing voices separation. We specify the problem definition in this dataset by categorizing it into i) unison, ii) duet, iii) main vs. rest, and iv) N-singing separation. Second, to overcome the absence of existing multi-singing datasets for a training purpose, we present a strategy for construction of multiple singing mixtures using various single-singing datasets. Third, we propose the improved super-resolution network (iSRNet), which greatly enhances initial estimates of separation networks. Jointly trained with the Conv-TasNet and the multi-singing mixture construction strategy, the proposed iSRNet achieved comparable performance to ideal time-frequency masks on duet and unison subsets of MedleyVox. Audio samples, the dataset, and codes are available on our website (https://github.com/jeonchangbin49/MedleyVox).
BibTeX
@inproceedings{jeon2023medleyvox,
  author = {Chang-Bin Jeon and Hyeongi Moon and Keunwoo Choi and Ben Sangbae Chon and Kyogu Lee},
  title = {{MedleyVox: An Evaluation Dataset for Multiple Singing Voices Separation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=SfJtkMQ0xd},
}
Figure