Abstract
This paper presents a novel framework for music source separation that leverages musical scores as weak supervision during training, eliminating the need for scarce isolated instrument tracks ("stems"). By employing a pre-trained transcription model as a "critic," the system learns to separate instruments from a mixture using score-derived guidance. Crucially, the score is only required during training; the resulting separator operates on raw audio alone during inference. We demonstrate that score-informed supervision provides vital frequency-domain information, significantly outperforming temporal-only weak labeling and bridging the gap toward fully supervised performance.
Introduction
Deep learning has revolutionized music source separation, but state-of-the-art models remain "data-hungry," requiring large datasets of isolated instrument tracks paired with their original mixtures. These datasets, such as MUSDB, are difficult to scale due to copyright restrictions and the labor-intensive process of obtaining clean stems.
While musical scores (MIDI files) are abundant online, previous "score-informed" methods usually required the score to be present during the actual separation process (inference). This paper investigates a different approach: Can we use scores as a weak label during training to build a separator that works on raw audio alone during testing?
Methods
The authors propose a three-step training strategy involving two main components: a Separator and a Transcriptor, both based on Temporal Convolutional Networks (TCNs).
Training Strategy
- Step 1: Transcriptor Training: A transcription model is trained to predict a multi-instrument "piano roll" from a mixture of audio, identifying which notes are played by which instruments and when.
- Step 2: Separator Training: The Transcriptor is frozen and acts as a critic. The Separator takes a mixture and predicts masks for each instrument. These separated outputs are fed into the Transcriptor. The Separator is penalized if the Transcriptor cannot accurately "read" the correct score from its separated audio. A Harmonic Mask Constraint is used to force the separator to focus on specific frequency bins where an instrument's harmonics should theoretically exist based on the score.
- Step 3: Joint Fine-Tuning with Adversarial Losses: To prevent the models from co-adapting to each other's mistakes, two adversarial losses are introduced: the Adversarial Mixture Loss ($L_{AML}$), which encourages the transcriptor to detect "fake" mixtures, and the Adversarial Transcription Loss ($L_{ATL}$), which forces sensitivity to residual noise.
Datasets and Experimental Setup
The study utilizes Slakh2100, a dataset synthesized from the Lakh MIDI Dataset. The authors focused on mixtures of acoustic piano, distorted electric guitar, and electric bass, totaling 145 hours of data. These instruments were chosen for their challenging, overlapping harmonic structures.
Key Results
Performance was primarily measured using the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR). The results demonstrate that incorporating score-based frequency information is superior to simple temporal activity labels.
- Baseline Comparison: The proposed system achieved an average SI-SDR of 5.8 dB, significantly outperforming a baseline weakly-supervised method (3.8 dB) that relied only on temporal activity labels.
- Ablation Gains: Adding the harmonic mask and adversarial losses provided incremental gains, particularly for the guitar, which is often the most difficult to isolate.
- The Performance Gap: While the score-supervised model is a significant step forward, a "fully supervised" model (trained with ground-truth stems) still leads at 8.2 dB, indicating room for further improvement.
- Transcription Bonus: Interestingly, fine-tuning the transcriptor alongside the separator improved transcription accuracy, suggesting a symbiotic relationship between the two tasks.
Discussion and Future Work
This work proves that "score labels" provide critical frequency-domain supervision that improves separation for instruments with overlapping ranges. By using a transcriptor as a critic, the model leverages the inherent relationship between separation and transcription: if a track cannot be transcribed, it hasn't been separated well enough.
However, limitations remain. The current model assumes perfect alignment between audio and MIDI, which is rarely the case with human performances. Future work will explore automated alignment techniques like Dynamic Time Warping. Additionally, the harmonic mask approach currently excludes non-harmonic instruments like drums, and expanding the framework to more diverse musical arrangements remains a key challenge.
Summary updated