MT3: Multi-Task Multitrack Music Transcription

Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, Jesse H. Engel

Automatic Music Transcription (AMT)—the process of converting raw audio into musical notation—is a cornerstone of computational music understanding. While speech recognition (ASR) has seen massive leaps in recent years, music remains a notoriously "low-resource" domain. Expert transcription is time-consuming, and existing datasets are tiny compared to the thousands of hours available for ASR or Natural Language Processing (NLP).

Historically, researchers have built specialized, handcrafted architectures for specific instruments (e.g., a model designed exclusively for piano). This approach fails to scale to "multitrack" scenarios where many instruments play at once, and it prevents models from sharing knowledge across different tasks. Furthermore, inconsistent evaluation metrics and dataset splits have made it difficult to track progress in the field.

To address these challenges, researchers have introduced MT3 (Multi-Task Multitrack Music Transcription), a unified framework that uses a single, general-purpose Transformer to transcribe arbitrary combinations of musical instruments from raw audio. By framing transcription as a sequence-to-sequence task and training on a mixture of datasets, MT3 achieves state-of-the-art results and provides a massive performance boost for low-resource instruments.

MT3 Framework

Methods: Treating Music Like Language

The authors propose a surprisingly simple solution: treat music transcription like a language translation problem. Instead of a complex, DSP-heavy pipeline with instrument-specific pitch-tracking heads, MT3 uses an off-the-shelf T5 (Text-to-Text Transfer Transformer) "small" model with approximately 60 million parameters.

  • Input: The model ingests log Mel spectrograms, a standard visual representation of audio frequencies over time.
  • Output: It generates a sequence of discrete "MIDI-like" tokens.

The Token Vocabulary

The core innovation enabling this architecture is a unified vocabulary that allows the model to describe any instrument's performance in a single stream.

Tokenization Process

The tokens include:

  • Instrument (128 values): Based on General MIDI programs (e.g., Piano, Cello, Trumpet).
  • Note (128 values): The pitch of the note.
  • On/Off: Indicates whether a note is starting or ending.
  • Time (205 values): Absolute time within a 2-second segment, quantized to 10ms.
  • Tie: A special token to handle notes that span across the boundaries of audio segments.

Multi-Task Training

The model is trained simultaneously on a "mixture" of six diverse multitrack AMT datasets: MAESTROv3 (solo piano), Slakh2100 (synthesized multitrack), Cerberus4 (guitar, bass, drums, piano), MusicNet (classical), GuitarSet (high-precision guitar), and URMP (multi-instrument classical).

To prevent the model from being overwhelmed by massive datasets (like Slakh2100) and ignoring smaller ones (like URMP), the authors employ temperature sampling. This strategy oversamples low-resource datasets during training, forcing the model to learn from them more effectively.

Key Insights and Results

  • The Power of Generality: A standard NLP architecture (T5) can outperform specialized music models. This suggests that the "inductive biases" previously thought necessary for music might be less important than large-scale, multi-task training.
  • Cross-Dataset Transfer: By training on a mixture, the model learns general features of music that apply across instruments. A model trained on a mix of piano and synthesized strings becomes significantly better at transcribing real strings in a low-resource dataset.
  • Massive Gains for Small Data: The mixture model showed incredible improvements on low-resource datasets compared to training on those datasets isolated. For example, on the URMP dataset, the Onset+Offset F1 score saw a 263% relative gain.
  • Unified Evaluation: The authors introduce Multi-instrument F1, a stricter metric requiring the model to get both the note and the instrument label correct. MT3 demonstrated high accuracy in identifying which instrument played which note, even in dense mixtures, exposing the limitations of previous "flat" models that ignored instrument identity.

Overall, MT3 achieved state-of-the-art results on all six datasets, often outperforming professional-grade DSP software.

Limitations and Future Work

While MT3 represents a significant leap forward, it has limitations. The model is currently restricted to the Western 12-tone equal temperament system and cannot handle microtonal music or non-Western traditions that use different pitch scales. Additionally, the authors noted that performance on datasets like MusicNet and URMP is likely capped by the quality of the ground-truth labels (such as timing misalignments) rather than the model's capacity.

Because expertly labeled music data is expensive to produce, the authors suggest that future work should explore self-supervised or semi-supervised learning on the vast amounts of unlabeled music audio available online, paving the way for even more robust and universal transcription models.

Summary updated

Abstract

Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneously, all while preserving fine-scale pitch and timing information. Further, many AMT datasets are ``low-resource'', as even expert musicians find music transcription difficult and time-consuming. Thus, prior work has focused on task-specific architectures, tailored to the individual instruments of each task. In this work, motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing (NLP), we demonstrate that a general-purpose Transformer model can perform multi-task AMT, jointly transcribing arbitrary combinations of musical instruments across several transcription datasets. We show this unified training framework achieves high-quality transcription results across a range of datasets, dramatically improving performance for low-resource instruments (such as guitar), while preserving strong performance for abundant instruments (such as piano). Finally, by expanding the scope of AMT, we expose the need for more consistent evaluation metrics and better dataset alignment, and provide a strong baseline for this new direction of multi-task AMT.
BibTeX
@inproceedings{gardner2022mt,
  author = {Josh Gardner and Ian Simon and Ethan Manilow and Curtis Hawthorne and Jesse H. Engel},
  title = {{MT3: Multi-Task Multitrack Music Transcription}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=cCYXI-m4-K},
}
Figure