MT3: Multi-Task Multitrack Music Transcription

Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, Jesse H. Engel

Abstract

Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneously, all while preserving fine-scale pitch and timing information. Further, many AMT datasets are "low-resource", as even expert musicians find music transcription difficult and time-consuming. Thus, prior work has focused on task-specific architectures, tailored to the individual instruments of each task. In this work, motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing (NLP), we demonstrate that a general-purpose Transformer model can perform multi-task AMT, jointly transcribing arbitrary combinations of musical instruments across several transcription datasets. We show this unified training framework achieves high-quality transcription results across a range of datasets, dramatically improving performance for low-resource instruments (such as guitar), while preserving strong performance for abundant instruments (such as piano). Finally, by expanding the scope of AMT, we expose the need for more consistent evaluation metrics and better dataset alignment, and provide a strong baseline for this new direction of multi-task AMT.

Introduction

Automatic Music Transcription (AMT) is the process of converting raw audio into symbolic notation, such as MIDI. It is a foundational challenge in music information retrieval, yet it remains significantly more complex than Automatic Speech Recognition (ASR). While ASR systems primarily disentangle the speech of a single individual, AMT requires systems to identify and separate multiple instruments playing concurrently, often with overlapping harmonic content, while maintaining millisecond-level precision for both pitch and timing.

Historically, the field has relied on "bespoke" models—architectures meticulously engineered for specific instruments or datasets. This fragmentation has hindered progress due to architectural rigidity, data scarcity for "low-resource" instruments, and inconsistent evaluation metrics across different studies.

The MT3 Framework

To address these challenges, the authors introduce MT3 (Multi-Task Multitrack Music Transcription). MT3 shifts away from specialized architectures in favor of a general-purpose Transformer model, specifically the T5 (Text-to-Text Transfer Transformer) "small" architecture (approximately 60 million parameters). By framing transcription as a sequence-to-sequence translation task, MT3 can transcribe arbitrary combinations of instruments from a single model.

Overview of the MT3 multi-task training framework showing inputs from various datasets and multitrack outputs.

The model processes log Mel spectrograms of 2-second audio segments. This unified approach allows the model to learn shared musical features across different datasets, effectively leveraging high-resource data to improve performance on rarer instruments.

Tokenization and Sequence-to-Sequence Modeling

A critical component of MT3 is its novel tokenization scheme, which converts musical events into a stream of "MIDI-like" tokens. This allows the Transformer to treat music transcription similarly to natural language translation.

The tokenization process converting piano rolls into a sequence of MIDI-like tokens including time, instrument program, and note events.

The output vocabulary includes:

  • Instrument (128 values): Based on General MIDI programs.
  • Note (128 values): Corresponding to MIDI pitches.
  • On/Off: Signaling the start and end of notes.
  • Time (205 values): Quantized to 10ms intervals within the segment.
  • Tie: A mechanism to handle notes that span across the 2-second segment boundaries, ensuring continuity.

Datasets and Multi-Task Training

MT3 is trained on a diverse mixture of six multitrack AMT datasets:

  1. MAESTROv3: Solo piano.
  2. Slakh2100: Large-scale synthetic multitrack data.
  3. Cerberus4: A Slakh subset focusing on common band instruments.
  4. MusicNet: Classical recordings with crowdsourced labels.
  5. GuitarSet: High-precision acoustic guitar recordings.
  6. URMP: Multi-instrument classical pieces recorded in isolation.

To balance these datasets, the authors use temperature sampling, up-sampling smaller datasets during training to prevent the model from being dominated by massive synthetic datasets like Slakh2100.

Results and Key Insights

MT3 achieves state-of-the-art (SOTA) results across all evaluated datasets. A key finding is the "mixture advantage": training on multiple datasets simultaneously acts as a powerful form of transfer learning. For example, on the low-resource URMP dataset, the multi-task model achieved a 260% relative gain in Onset+Offset F1 score compared to a model trained on URMP alone.

The model demonstrates remarkable robustness, maintaining high performance on solo piano while accurately assigning notes to the correct instruments in complex orchestral or band arrangements. The authors introduced a "Multi-instrument F1" metric to specifically evaluate this capability, where MT3 showed strong performance in identifying both the correct note and the correct instrument.

Conclusion and Future Work

MT3 demonstrates that a general-purpose Transformer can outperform specialized music architectures by leveraging multi-task learning. However, limitations remain. The current model is tied to the 12-tone equal temperament system and does not yet capture expressive nuances like pitch bends or velocity (volume). Future research may explore self-supervised learning on large unlabeled music libraries to further enhance the model's capabilities and generalization.

Summary updated

Abstract

Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneously, all while preserving fine-scale pitch and timing information. Further, many AMT datasets are "low-resource", as even expert musicians find music transcription difficult and time-consuming. Thus, prior work has focused on task-specific architectures, tailored to the individual instruments of each task. In this work, motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing (NLP), we demonstrate that a general-purpose Transformer model can perform multi-task AMT, jointly transcribing arbitrary combinations of musical instruments across several transcription datasets. We show this unified training framework achieves high-quality transcription results across a range of datasets, dramatically improving performance for low-resource instruments (such as guitar), while preserving strong performance for abundant instruments (such as piano). Finally, by expanding the scope of AMT, we expose the need for more consistent evaluation metrics and better dataset alignment, and provide a strong baseline for this new direction of multi-task AMT.
BibTeX
@inproceedings{gardner2022mt,
  author = {Josh Gardner and Ian Simon and Ethan Manilow and Curtis Hawthorne and Jesse H. Engel},
  title = {{MT3: Multi-Task Multitrack Music Transcription}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=cCYXI-m4-K},
}
Figure