Sample-efficient end-to-end Optical Music Recognition through Active Learning

Eric Ayllon, Francisco J. Castellanos, Antonio Rios-VilaarXiv 2026 · arXiv

TL;DR

This paper presents the first systematic study of Active Learning (AL) for Optical Music Recognition (OMR), demonstrating that fine-tuning a pre-trained model with as few as 10 carefully selected samples can drastically improve performance on new datasets. The authors find that the common practice of selecting the "most uncertain" samples isn't always optimal; instead, selecting moderately uncertain or even relatively certain samples often yields better results by avoiding noisy outliers.

What Problem Does This Paper Address?

Optical Music Recognition (OMR) is the "OCR for music"—the process of turning scanned sheet music into machine-readable formats. While deep learning has pushed OMR toward "end-to-end" models that skip messy manual layout analysis, these models are notoriously brittle. A model trained on printed 19th-century scores often fails miserably when faced with 17th-century handwritten manuscripts.

The standard fix is fine-tuning: take a pre-trained model and train it further on a small slice of the new data. However, annotating music is incredibly labor-intensive, requiring specialized knowledge. This paper asks: If we only have the budget to annotate a handful of pages, which ones should we pick to get the biggest performance boost?

Methods

The authors evaluate two state-of-the-art OMR architectures:

  1. CRNN (Convolutional Recurrent Neural Network): Used for staff-level transcription (processing one line of music at a time).
  2. SMT (Sheet Music Transformer): A more modern, "LA-free" (Layout Analysis-free) approach that transcribes entire pages at once.

To select the best samples for annotation, they compared several Active Learning (AL) strategies:

  • Uncertainty-based Sampling: The model "guesses" on unlabeled data and we pick samples based on its confidence. They tested extremes (Highest/Lowest uncertainty) and middle-ground heuristics (Near the mean, or specific quartiles like $Q_1$ and $Q_3$).
  • Diversity-based Sampling: Using the model's internal embeddings to cluster the data and pick representative samples from across the entire distribution, ensuring the training set isn't redundant.
  • Baselines: Random sampling and an "Oracle" (which picks samples based on the actual known error, a "perfect" but unrealistic scenario).

Datasets

The models were pre-trained on large synthetic datasets (PrIMuS for staves and GrandStaff for full pages) and then evaluated on several real-world collections:

  • Staff-level (Handwritten & Printed): Capitan, Guatemala, Magnificat, Mottecta, and Seils. These range from 16th-century Spanish choir books to Italian madrigals.
  • Full-page (Printed): Mozarteum (Mozart's piano works) and Polish Scores.

Key Insights and Ideas

  • The "Highest Uncertainty" Trap: In many AI fields, the go-to strategy is to label the samples the model is most confused about. This paper finds that in OMR, these are often "outliers" or noisy images that don't help the model generalize.
  • Architecture Matters for Strategy:
    • For staff-level (CRNN) models, picking samples near the third quartile ($Q_3$) of uncertainty worked best. These are "challenging but solvable" samples.
  • For full-page (SMT) models, picking samples near the first quartile ($Q_1$)—samples the model is relatively confident about—actually performed better. The authors suggest this is because Transformers are data-hungry and benefit more from a "stable" training set that reinforces correct patterns.
  • Logarithmic Returns: The jump from 0 to 10 samples is massive. The jump from 50 to 100 is much smaller. This suggests that for researchers with limited budgets, a "one-shot" active learning cycle is highly efficient.

Key Results

The quantitative gains are striking. Using the Symbol Error Rate (SER) as the primary metric:

  • Minimal Data, Maximum Impact: On the Capitan dataset, the zero-shot (no fine-tuning) error was 95.7%. Labeling just 10 samples dropped that error to 55.6%. By 100 samples, it hit 21.5%.
  • Efficiency: For staff-level collections, using just 100 samples achieved an average error reduction of over 80% relative to the "Oracle" performance (training on the entire dataset).
  • Confidence as a Proxy: Confidence-based sampling performed nearly as well as the "Oracle" selection, proving that model confidence is a reliable signal for data selection in OMR.

Limitations and Future Work

The authors acknowledge a few key constraints:

  • One-Shot Learning: The study focused on a single round of selection and fine-tuning. In practice, multiple iterative rounds of AL might yield even better results but were excluded here to maintain a controlled experimental setup.
  • Computational Cost: Diversity-based sampling requires extracting embeddings and clustering, which is significantly more computationally expensive than simple uncertainty ranking.
  • Domain Specifics: While the strategies are general, the "best" quartile for uncertainty seems to depend on the specific model architecture (CRNN vs. Transformer), suggesting there is no "one size fits all" strategy for all OMR tasks.

Future work could involve combining AL with self-supervised learning to better leverage the vast amounts of unlabeled music data available in digital libraries.

Summary updated

Abstract

End-to-end approaches to Optical Music Recognition (OMR) have recently shown promising results, but their performance often degrades on unseen data. A common solution is to pretrain and fine-tune models with annotated samples from the target domain, improving adaptation to new distributions. However, this process typically requires extensive manual annotation to yield competitive models, making it costly and time-consuming. In this work, we explore Active Learning to reduce this burden by selecting informative samples for labeling in low-resource scenarios, where the amount of manual annotation that can be performed is limited. We evaluate both uncertainty- and diversity-based sampling strategies on staff-level and full-page corpora. Results show that confidence-based sampling performs comparably to oracle selection (based on true error), and that each scenario benefits from a tailored strategy.
BibTeX
@misc{ayllon2026sample,
  author = {Eric Ayllon and Francisco J. Castellanos and Antonio Rios-Vila},
  title = {{Sample-efficient end-to-end Optical Music Recognition through Active Learning}},
  year = {2026},
  eprint = {2611.25975},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25975},
}
Figure