Unaligned Supervision for Automatic Music Transcription in The Wild

Ben Maman, Amit H. Bermano

Introduction

Automatic Music Transcription (AMT)—the process of converting raw audio into digital sheet music—is a long-standing challenge in music technology. While deep neural networks have shown promise, they are severely limited by a "data bottleneck." Creating ground-truth labels requires millisecond-precise timing for every note, a task so difficult that high-quality datasets have been largely restricted to piano music recorded on specialized digital instruments.

NoteEM is a new framework designed to break this bottleneck. It allows models to learn transcription from "in-the-wild" recordings using only unaligned musical scores (MIDI files) as supervision, effectively opening the door to high-accuracy transcription for any instrument.

The NoteEM Framework

NoteEM treats transcription as an Expectation-Maximization (EM) problem. Instead of requiring perfectly timed labels, the system iteratively improves both its transcription and its alignment of the available scores.

The NoteEM Expectation-Maximization pipeline

As shown in the pipeline above, the process works in three main stages:

  1. Bootstrapping: The model is first trained on purely synthetic audio (MIDI files rendered via software) to gain a basic understanding of musical structures.
  2. The E-Step (Alignment): The model takes an unlabeled real-world recording and predicts note probabilities (logits). It then uses Dynamic Time Warping (DTW) to align the unaligned musical score to these predictions rather than raw audio features.
  3. The M-Step (Training): These newly aligned labels are used as "ground truth" to retrain and refine the model.

To ensure robustness, the framework employs pseudo-labeling to account for performance variations and pitch-shift equivariance to regularize the model by ensuring that shifts in audio pitch result in corresponding shifts in transcribed notes.

Experimental Setup and Datasets

The study utilizes a diverse range of data to prove the framework's versatility:

  • Standard Benchmarks: MAESTRO and MAPS (piano) were used for rigorous evaluation.
  • Multi-Instrument Data: The authors utilized MusicNet and even provided a new set of labels (MusicNetEM) that they claim are more accurate than the original manual annotations.
  • Generalization: GuitarSet and a 74-hour self-collected dataset (including orchestra, harpsichord, and lute) were used to demonstrate how the framework scales to unseen instruments and real-world environments.

Key Results

NoteEM achieves state-of-the-art (SOTA) accuracy across multiple benchmarks, often surpassing models trained with full, manual supervision.

Performance comparison of NoteEM against supervised and weakly-supervised baselines on MAESTRO and MAPS datasets.

Key performance highlights include:

  • SOTA on Piano: On the MAPS dataset, NoteEM achieved a note-level F1 score of 87.3%, surpassing previous fully supervised models (86.4%).
  • Cross-Dataset Mastery: When trained on synthetic data and evaluated on the MAESTRO test set, NoteEM reached an 89.7% note-level F1, a significant improvement over previous weakly-supervised attempts that struggled to exceed 30%.
  • Multi-Instrument Success: The model successfully transcribed 11-22 different instruments, showing marked improvements for challenging instruments like the horn, bassoon, and clarinet.

Insights and Future Work

A core insight of this research is that data variety (different instruments and room acoustics) is often more important for model generalization than "perfect" labels from a narrow source. By using the model as its own teacher, the system creates a virtuous cycle where better transcription leads to tighter alignment, which in turn further improves the transcriber.

While NoteEM excels at note onsets, challenges remain in capturing precise note "offsets" (endings) and "velocity" (dynamics), as these are often obscured by acoustic environments. Future work may extend this framework to vocal transcription or use its high-quality outputs to condition generative AI models for music synthesis.

Summary updated

Abstract

Multi-instrument Automatic Music Transcription (AMT), or the decoding of a musical recording into semantic musical content, is one of the holy grails of Music Information Retrieval. Current AMT approaches are restricted to piano and (some) guitar recordings, due to difficult data collection. In order to overcome data collection barriers, previous AMT approaches attempt to employ musical scores in the form of a digitized version of the same song or piece. The scores are typically aligned using audio features and strenuous human intervention to generate training labels. We introduce Note$_{EM}$, a method for simultaneously training a transcriber and aligning the scores to their corresponding performances, in a fully-automated process. Using this unaligned supervision scheme, complemented by pseudo-labels and pitch shift augmentation, our method enables training on in-the-wild recordings with unprecedented accuracy and instrumental variety. Using only synthetic data and unaligned supervision, we report SOTA note-level accuracy of the MAPS dataset, and large favorable margins on cross-dataset evaluations. We also demonstrate robustness and ease of use; we report comparable results when training on a small, easily obtainable, self-collected dataset, and we propose alternative labeling to the MusicNet dataset, which we show to be more accurate. Our project page is available at https://benadar293.github.io.
BibTeX
@inproceedings{maman2022unaligned,
  author = {Ben Maman and Amit H. Bermano},
  title = {{Unaligned Supervision for Automatic Music Transcription in The Wild}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=PgnHNe0NjS},
}
Figure