Self-Refining of Pseudo Labels for Music Source Separation With Noisy Labeled Data

Junghyun Koo, Yunkee Chae, Chang-Bin Jeon, Kyogu Lee

Introduction

Music Source Separation (MSS) is the task of isolating individual instrument tracks—such as vocals, drums, and bass—from a complete musical mixture. The state-of-the-art in MSS relies heavily on supervised deep learning, which requires vast amounts of high-quality "stems" (isolated recordings). While the internet provides a nearly infinite supply of audio data, these sources are frequently "noisy": a track labeled as "bass" might contain significant bleeding from other instruments, or a file might be entirely mislabeled due to human error.

Standard MSS models are highly sensitive to these discrepancies. When trained on corrupted ground truth, models learn to reproduce those errors, leading to "dirty" separations filled with artifacts. This paper introduces an automated "self-refining" technique designed to overcome this bottleneck by using a multi-label classifier to iteratively clean its own training labels, allowing researchers to leverage massive, uncurated web datasets effectively.

The Self-Refining Framework

The authors propose a two-stage pipeline that transforms a noisy dataset into a refined training set for source separation models.

1. Robust Instrument Classifier Training

The process begins by training a multi-label instrument classifier based on the ConvNeXt architecture. To make this classifier resilient to the inherent noise in the initial dataset, the authors employ a random mixing technique. By taking noisy stems and mixing them together to create synthetic mixtures with associated pseudo-labels, the model is forced to learn the distinct acoustic characteristics of instruments even when the source labels are imperfect.

2. Dataset Refinement and MSS Training

Once trained, the classifier is used to "re-label" the original noisy dataset. It identifies which instruments are actually present in each track, effectively filtering out mislabeled data or identifying tracks with significant bleed. The authors found that an iterative approach—where the refined labels from one round are used to train a better classifier for the next—yields the best results, though performance typically plateaus after two iterations.

Finally, standard source separation models, such as Hybrid Demucs or X-UMX, are trained using this refined dataset rather than the original noisy metadata.

Experimental Setup and Datasets

The methodology was validated using two primary datasets:

  • MDX2023: A dataset specifically designed for the Music Demixing Challenge 2023, containing 203 songs with intentional corruptions, including bleeding and labeling errors.
  • MUSDB18: The industry-standard "clean" dataset consisting of 150 professionally curated songs. This served as the gold standard for evaluating the upper-bound performance of the self-refining method.

Key Insights

The "Noisy-Refined" Paradox

One of the most surprising findings was that an MSS model trained on data refined by a "noisy" classifier (one trained on the corrupted MDX2023 data) performed better than one refined by a classifier trained on the "clean" MUSDB18 data. This suggests that a classifier trained on the specific noise distribution of a dataset is better equipped to recognize and correct the specific types of errors present in that environment.

The Importance of Recall

In the context of source separation, "False Negatives" (failing to label an instrument that is actually present) are significantly more damaging than "False Positives." If a classifier fails to detect a vocal bleed in a drum track, the MSS model will attempt to force that vocal sound into the drum output, creating audible artifacts. High recall in the refinement stage is therefore critical for maintaining separation quality.

Results

The self-refining method demonstrated significant improvements across all metrics:

  • Instrument Recognition: The self-refined classifier achieved 89.2% accuracy on multi-label recognition, trailing the "clean" gold standard (90.2%) by only 1%.
  • Separation Quality: Using the X-UMX model, the average Signal-to-Distortion Ratio (SDR) improved from a baseline of 1.91 dB (trained on raw noisy data) to 4.28 dB (trained on self-refined data). For context, the same model trained on the perfectly clean MUSDB18 dataset achieved 4.83 dB.

These results indicate that the self-refining process can bridge the gap between noisy web-scraped data and expensive, professionally curated datasets.

Conclusion and Future Work

This research provides a scalable solution for the data bottleneck in Music Source Separation. By automating the cleaning of noisy labels, the self-refining technique allows models to ignore metadata errors and focus on the underlying acoustic signals. While the computational overhead of training an auxiliary classifier is a factor, the gains in separation quality are substantial.

Future research may explore the limits of this method—specifically how it handles datasets where more than 50% of the labels are corrupted—and its potential application to other Music Information Retrieval (MIR) tasks like automatic transcription or beat tracking, where large-scale, messy data is prevalent.

Summary updated

Abstract

Music source separation (MSS) faces challenges due to limited availability and potential noise in correctly labeled individual instrument tracks. In this paper, we propose an automated approach for refining mislabeled instrument tracks in a partially noisy-labeled dataset. The proposed self-refining technique with noisy-labeled dataset results in only a 1% accuracy degradation for multi-label instrument recognition compared to a classifier trained with a clean-labeled dataset. The study demonstrates the importance of refining noisy-labeled data for training MSS models and shows that utilizing the refined dataset for MSS leads to comparable results to a clean-labeled dataset. Notably, upon only access to a noisy dataset, MSS models trained on self-refined datasets even outperformed those trained on datasets refined with a classifier trained on clean labels.
BibTeX
@inproceedings{koo2023self,
  author = {Junghyun Koo and Yunkee Chae and Chang-Bin Jeon and Kyogu Lee},
  title = {{Self-Refining of Pseudo Labels for Music Source Separation With Noisy Labeled Data}},
  booktitle = {ISMIR},
  year = {2023},
  url = {https://openreview.net/forum?id=BftQKwZwSI},
}
Figure