Transcription Is All You Need: Learning To Separate Musical Mixtures With Score As Supervision

Yun-Ning Hung, Gordon Wichern, Jonathan Le Roux

Abstract

This paper presents a novel framework for music source separation that leverages musical scores as weak supervision during training, eliminating the need for scarce isolated instrument tracks ("stems"). By employing a pre-trained transcription model as a "critic," the system learns to separate instruments from a mixture using score-derived guidance. Crucially, the score is only required during training; the resulting separator operates on raw audio alone during inference. We demonstrate that score-informed supervision provides vital frequency-domain information, significantly outperforming temporal-only weak labeling and bridging the gap toward fully supervised performance.

Introduction

Deep learning has revolutionized music source separation, but state-of-the-art models remain "data-hungry," requiring large datasets of isolated instrument tracks paired with their original mixtures. These datasets, such as MUSDB, are difficult to scale due to copyright restrictions and the labor-intensive process of obtaining clean stems.

While musical scores (MIDI files) are abundant online, previous "score-informed" methods usually required the score to be present during the actual separation process (inference). This paper investigates a different approach: Can we use scores as a weak label during training to build a separator that works on raw audio alone during testing?

Methods

The authors propose a three-step training strategy involving two main components: a Separator and a Transcriptor, both based on Temporal Convolutional Networks (TCNs).

The proposed three-step training framework involving Transcriptor and Separator components with score-based losses.

Training Strategy

  1. Step 1: Transcriptor Training: A transcription model is trained to predict a multi-instrument "piano roll" from a mixture of audio, identifying which notes are played by which instruments and when.
  2. Step 2: Separator Training: The Transcriptor is frozen and acts as a critic. The Separator takes a mixture and predicts masks for each instrument. These separated outputs are fed into the Transcriptor. The Separator is penalized if the Transcriptor cannot accurately "read" the correct score from its separated audio. A Harmonic Mask Constraint is used to force the separator to focus on specific frequency bins where an instrument's harmonics should theoretically exist based on the score.
  3. Step 3: Joint Fine-Tuning with Adversarial Losses: To prevent the models from co-adapting to each other's mistakes, two adversarial losses are introduced: the Adversarial Mixture Loss ($L_{AML}$), which encourages the transcriptor to detect "fake" mixtures, and the Adversarial Transcription Loss ($L_{ATL}$), which forces sensitivity to residual noise.

Datasets and Experimental Setup

The study utilizes Slakh2100, a dataset synthesized from the Lakh MIDI Dataset. The authors focused on mixtures of acoustic piano, distorted electric guitar, and electric bass, totaling 145 hours of data. These instruments were chosen for their challenging, overlapping harmonic structures.

Key Results

Performance was primarily measured using the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR). The results demonstrate that incorporating score-based frequency information is superior to simple temporal activity labels.

SI-SDR results comparing the proposed weakly-supervised method against supervised and baseline models.
  • Baseline Comparison: The proposed system achieved an average SI-SDR of 5.8 dB, significantly outperforming a baseline weakly-supervised method (3.8 dB) that relied only on temporal activity labels.
  • Ablation Gains: Adding the harmonic mask and adversarial losses provided incremental gains, particularly for the guitar, which is often the most difficult to isolate.
  • The Performance Gap: While the score-supervised model is a significant step forward, a "fully supervised" model (trained with ground-truth stems) still leads at 8.2 dB, indicating room for further improvement.
  • Transcription Bonus: Interestingly, fine-tuning the transcriptor alongside the separator improved transcription accuracy, suggesting a symbiotic relationship between the two tasks.

Discussion and Future Work

This work proves that "score labels" provide critical frequency-domain supervision that improves separation for instruments with overlapping ranges. By using a transcriptor as a critic, the model leverages the inherent relationship between separation and transcription: if a track cannot be transcribed, it hasn't been separated well enough.

However, limitations remain. The current model assumes perfect alignment between audio and MIDI, which is rarely the case with human performances. Future work will explore automated alignment techniques like Dynamic Time Warping. Additionally, the harmonic mask approach currently excludes non-harmonic instruments like drums, and expanding the framework to more diverse musical arrangements remains a key challenge.

Summary updated

Abstract

Most music source separation systems require large collections of isolated sources for training, which can be difficult to obtain. In this work, we use musical scores, which are comparatively easy to obtain, as a weak label for training a source separation system. In contrast with previous score-informed separation approaches, our system does not require isolated sources, and score is used only as a training target, not required for inference. Our model consists of a separator that outputs a time-frequency mask for each instrument, and a transcriptor that acts as a critic, providing both temporal and frequency supervision to guide the learning of the separator. A harmonic mask constraint is introduced as another way of leveraging score information during training, and we propose two novel adversarial losses for additional fine-tuning of both the transcriptor and the separator. Results demonstrate that using score information outper-forms temporal weak-labels, and adversarial structures lead to further improvements in both separation and transcription performance.
BibTeX
@inproceedings{hung2021transcription,
  author = {Yun-Ning Hung and Gordon Wichern and Jonathan Le Roux},
  title = {{Transcription Is All You Need: Learning To Separate Musical Mixtures With Score As Supervision}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413358},
}
Figure