MR-MT3: Memory Retaining Multi-Track Music Transcription to Mitigate Instrument Leakage

Hao Hao Tan, Kin Wai Cheuk, Taemin Cho, Wei-Hsiang Liao, Yuki MitsufujiarXiv 2024 · DOI · arXiv

Introduction

Automatic Music Transcription (AMT) is often considered the "holy grail" of music technology, aiming to convert raw audio into symbolic notation like MIDI. While single-instrument transcription (e.g., solo piano) is largely a solved problem, multi-instrument transcription remains a massive challenge due to the complexity of overlapping sounds and instrument identification.

The Challenge: Instrument Leakage

The current leading model, MT3, treats transcription as a sequence-to-sequence task, converting spectrograms into "event tokens." However, MT3 processes audio in short, independent segments. Because it lacks long-term memory, it often loses track of which instrument is playing what across segment boundaries. This leads to instrument leakage, a common error where a melody might start on a "Piano" track in one segment but "leak" into a "Guitar" track in the next. The result is a fragmented MIDI file that requires extensive manual cleanup.

Visualizing instrument leakage: overprediction vs. ground truth.

Methods: Introducing Memory Retention

Researchers from BandLab and Sony AI proposed MR-MT3 (Memory Retaining MT3), which introduces three key innovations to solve the continuity problem:

  1. Memory Retention Mechanism: The model now includes a dedicated memory module. It takes the tokens generated from the previous audio segment, processes them through a self-attention layer, and feeds that "summary" into the current segment's decoding process. This provides the model with the musical context of what was just playing.
  2. Prior Token Sampling: During training, instead of always looking at the immediate previous segment, the model is trained to sample from various points in the recent past. This makes the memory mechanism more robust and flexible.
  3. Token Shuffling: MT3 usually predicts tokens in a strict, fixed order. The authors introduced token shuffling as a data augmentation technique, forcing the model to learn underlying musical relationships rather than just memorizing a sequence pattern.
The MR-MT3 architecture featuring the integrated memory module.

Key Results and Insights

The improvements were both quantitative and qualitative, evaluated on benchmarks like Slakh2100 and ComMU:

  • Reduced Leakage: The Instrument Leakage Ratio ($\phi$)—a metric introduced to quantify "hallucinated" instruments—dropped from 1.70 (heavy over-prediction) to 1.12 (much closer to the ground truth of 1.0).
  • Better Identification: The Instrument Detection F1 score improved from roughly 34% to 41%, meaning the model is significantly better at identifying the correct ensemble members.
  • Cleaner MIDI: Qualitatively, the transcriptions are much less fragmented. Instead of a melody being scattered across multiple tracks, it stays consistently within the correct instrument track.

One surprising finding was the "Complexity Paradox": models trained on complex multi-instrument mixes often perform poorly on simple solo tracks because they become conditioned to expect complexity and hallucinate extra instruments. MR-MT3's memory and shuffling mechanisms help mitigate this by providing better contextual grounding.

Conclusion

MR-MT3 demonstrates that instrument identification is not just about the timbre of a single note, but the continuity of the performance. By adding memory, the model realizes that if a violin was playing a second ago, a subsequent high-pitched note is likely still that violin. Future work will focus on finding a "universal" architecture that handles solo instruments and full orchestras with equal grace.

Summary updated

Abstract

This paper presents enhancements to the MT3 model, a state-of-the-art (SOTA) token-based multi-instrument automatic music transcription (AMT) model. Despite SOTA performance, MT3 has the issue of instrument leakage, where transcriptions are fragmented across different instruments. To mitigate this, we propose MR-MT3, with enhancements including a memory retention mechanism, prior token sampling, and token shuffling are proposed. These methods are evaluated on the Slakh2100 dataset, demonstrating improved onset F1 scores and reduced instrument leakage. In addition to the conventional multi-instrument transcription F1 score, new metrics such as the instrument leakage ratio and the instrument detection F1 score are introduced for a more comprehensive assessment of transcription quality. The study also explores the issue of domain overfitting by evaluating MT3 on single-instrument monophonic datasets such as ComMU and NSynth. The findings, along with the source code, are shared to facilitate future work aimed at refining token-based multi-instrument AMT models.
BibTeX
@misc{tan2024mr,
  author = {Hao Hao Tan and Kin Wai Cheuk and Taemin Cho and Wei-Hsiang Liao and Yuki Mitsufuji},
  title = {{MR-MT3: Memory Retaining Multi-Track Music Transcription to Mitigate Instrument Leakage}},
  year = {2024},
  eprint = {2403.10024},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.10024},
}
Figure