Transcription Is All You Need: Learning to Separate Musical Mixtures with Score as Supervision

Yun-Ning Hung, Gordon Wichern, Jonathan Le Roux

TL;DR

Researchers have developed a way to train music source separation models—systems that pull apart individual instruments from a song—using only musical scores (like MIDI files) as supervision. By using a pre-trained transcription model as a "critic" to guide the separator, this approach eliminates the need for hard-to-find isolated instrument tracks (stems) during training, while requiring only the raw audio mixture at inference time.

What Problem Does This Paper Address?

Most state-of-the-art music source separation systems rely on supervised learning, which requires massive datasets of "stems"—isolated recordings of each instrument in a mix. Unfortunately, stems are notoriously difficult to obtain due to copyright restrictions and the labor-intensive nature of recording them. Existing open datasets like MUSDB are relatively small compared to the vast amount of music available.

Previous attempts at "weakly-supervised" separation used simple temporal labels (e.g., "the guitar is playing now"). However, music is complex; instruments often play simultaneously for the entire song, and their harmonic structures overlap in frequency. Temporal labels alone aren't enough to tell a model how to distinguish a bass note from a piano note when they occupy the same time and frequency space.

Methods

The authors propose a three-step training strategy that leverages the abundance of musical scores (which are much easier to find or create than stems) as a source of rich frequency and temporal information.

![Fig.

  1. The three-step training strategy: 1) Train transcriptor, 2) Train separator with transcriptor as critic, 3) Joint fine-tuning.](/figures/art_fd82a4b7afd14d3989c47df093d4313d)
  2. Step 1: Transcriptor Training: They first train a Transcriptor (based on a Temporal Convolutional Network or TCN) to predict a multi-instrument piano roll from a music mixture.
  3. Step 2: Separator Training: They introduce a Separator that generates time-frequency masks for each instrument. Crucially, the pre-trained Transcriptor acts as a critic. The Separator is rewarded if the Transcriptor can accurately "read" the score from the Separator's output. If the Separator leaks piano sounds into the guitar track, the Transcriptor will fail to find the correct guitar score, and the model will adjust.
  4. Step 3: Joint Fine-Tuning & Adversarial Loss: Finally, both models are trained together. To prevent the Separator from finding "loopholes" (tricking a fixed Transcriptor), they introduce adversarial losses. For example, they create "fake" mixtures by combining separated tracks from different songs and challenge the Transcriptor to identify them as incorrect.

They also introduce a Harmonic Mask Mixture Loss ($\mathcal{L}_{h-mix}$). By using the score to calculate where an instrument's harmonics should be, they can force the separator to focus energy on those specific frequency bins.

Datasets

The study utilizes the Slakh2100 dataset, which contains 145 hours of mono audio synthesized from the Lakh MIDI Dataset using professional-grade virtual instruments. This provides perfectly aligned audio and MIDI scores. For these experiments, the authors focused on mixtures of acoustic piano, distorted electric guitar, and electric bass.

Key Insights and Ideas

  • Transcription as a Richer Signal: Unlike simple binary activity labels, a musical score provides a precise map of both when and at what frequency an instrument should be active.
  • The Critic Approach: Using a transcriptor as a critic allows the model to learn separation without ever seeing a "clean" isolated instrument track. The "ground truth" is the abstract score, not the audio itself.
  • Adversarial Regularization: The authors found that if the transcriptor is too "robust," the separator stops improving. Adversarial training forces the transcriptor to become more sensitive to separation errors, which in turn pushes the separator to produce cleaner results.
  • Harmonic Constraints: Incorporating domain knowledge about how instruments produce sound (harmonics) significantly improves the separation of instruments with overlapping ranges, like piano and guitar.

Key Results

The proposed method significantly outperformed baseline weakly-supervised methods that only use temporal labels.

Table 1. Separation performance in terms of SI-SDR (dB), where ‘isolated’ and ‘fine-tune’ respectively indicate the training setup of step 2 and step 3.

As shown in Table 1, the best weakly-supervised configuration achieved an average SI-SDR of 5.8 dB, compared to just 3.8 dB for the baseline. While this still trails fully supervised models (8.2 dB), it closes a significant portion of the gap using only scores—data that is vastly more accessible than isolated stems.

Limitations and Future Work

  • Alignment: The current work assumes the audio and score are perfectly aligned (a byproduct of using synthesized data). In the real world, human performances vary in tempo. Future work will need to integrate automated alignment (like Dynamic Time Warping).
  • Non-Harmonic Instruments: The current harmonic mask approach doesn't apply to percussive instruments like drums.
  • Instrument Similarity: The model still struggles to distinguish between instruments with very similar timbres (e.g., different types of electric guitars).
  • Semi-Supervised Potential: The authors suggest that combining a small amount of supervised data with a large amount of score-only data could be a powerful next step.

Summary updated

Abstract

Most music source separation systems require large collections of isolated sources for training, which can be difficult to obtain. In this work, we use musical scores, which are comparatively easy to obtain, as a weak label for training a source separation system. In contrast with previous score-informed separation approaches, our system does not require isolated sources, and score is used only as a training target, not required for inference. Our model consists of a separator that outputs a time-frequency mask for each instrument, and a transcriptor that acts as a critic, providing both temporal and frequency supervision to guide the learning of the separator. A harmonic mask constraint is introduced as another way of leveraging score information during training, and we propose two novel adversarial losses for additional fine-tuning of both the transcriptor and the separator. Results demonstrate that using score information outperforms temporal weak-labels, and adversarial structures lead to further improvements in both separation and transcription performance.
BibTeX
@inproceedings{hung2021transcription,
  author = {Yun-Ning Hung and Gordon Wichern and Jonathan Le Roux},
  title = {{Transcription Is All You Need: Learning to Separate Musical Mixtures with Score as Supervision}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413358},
}
Figure