Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning

Zhilin Wang, Zhe Yang, Yun Luo, Yafu Li, Xiaoye Qu, Ziqian Qiao, Haoran Zhang, Runzhe Zhan, Derek F. Wong, Ji-Zhe Zhou, Yu ChengSubmission to ICLR 2026 · Rejected

TL;DR

By treating the deterministic rules of music theory like mathematical operators, the authors programmatically synthesize thousands of verifiable sheet music reasoning problems across textual (ABC notation) and visual (score image) modalities. Training open-source language and vision models on this synthetic data using Reinforcement Learning with Verifiable Rewards (RLVR) yields dramatic performance gains on sheet music benchmarks, improves downstream composition, and surprisingly boosts performance on standard mathematical reasoning tasks.

What Problem Does This Paper Address?

While Large Language Models (LLMs) and Multimodal LLMs (MLLMs) have made remarkable strides in math, coding, and general knowledge, they often fail at sheet music reasoning—the ability to interpret and apply music theory principles (such as rhythmic summation, interval computation, or chord construction) to written notation.

Existing benchmarks (e.g., MusicTheoryBench, MMMU Music) rely on scarce, human-crafted questions. This manual pipeline creates a severe data bottleneck: it cannot supply the thousands of verifiable, scalable problem–answer pairs required for modern reinforcement learning methods like RLVR.

Methods

The authors introduce a fully programmatic data synthesis framework that operates without LLMs in the loop, ensuring rigorous verification:

  • Rule-Based Question Templates: Core music theory principles are mapped into nine programmatic question classes across four foundational domains: Rhythm, Chord, Interval, and Scale.
  • Dual-Modality Pipeline: Questions and seed scores (sampled from the ABC notation dataset MelodyHub) are generated first as text. Tools like abcm2ps and ImageMagick then render the notation into staff notation images for Visual Question Answering (VQA).
  • Reinforcement Learning with Verifiable Rewards (RLVR): Using Group Relative Policy Optimization (GRPO), models are trained against an exact-match accuracy reward on 8,000 synthetic questions per modality without format-specific penalty terms.
ABC Score / Music Theory Rules ➔ Programmatic Question Class ➔ Textual QA (ABC)
                                                                     ↓ (Render)
                                                              Visual QA (Image)

Datasets

  • SSMR-Bench (Introduced): An evaluation benchmark containing 1,600 textual QA and 1,600 visual QA pairs, split evenly (25% each) across Rhythm, Chord, Interval, and Scale.
  • SSMR-Train (Introduced): A training set consisting of 8,000 textual and 8,000 visual verifiable problems.
  • External Evaluation Datasets: Used to test out-of-distribution transfer:
  • MusicTheoryBench (human-curated textual music theory)
  • MMMU (Music subset) (college-level visual music reasoning)
  • Math benchmarks (AIME24, AMC, MATH-500, Minerva, Olympiad Bench)

Key Insights and Ideas

  • Music as Math: Musical structures (fractional time signatures, pitch semitone distances, tertian chord stacks) obey strict formal logic. They can be programmatically verified just like arithmetic.
  • The Visual Recognition Bottleneck: Frontier vision-language models struggle far more with sheet music images than text. For instance, Gemini 2.5 Pro scores 96.19% on textual SSMR-Bench questions but collapses to 55.44% on the exact same questions presented as images. Visual symbol decoding acts as a steep barrier before reasoning can even occur.
  • Cross-Domain Generalization (Music $\to$ Math): Training Qwen3-8B-Base solely on synthetic music reasoning data improves its average mathematical benchmark score from 33.90% to 43.97% (including MATH-500 jumping from 66.00% to 79.40%). This confirms that optimizing for symbolic constraint-satisfaction and fractional summation elicits generalizable reasoning patterns.
  • Reasoning Boosts Musical Composition: Improving symbolic reasoning directly improves four-measure melodic continuation, reducing rhythmic errors and improving harmonic coherence.

Key Results

  • SSMR-Bench Textual QA: Frontier models lead the board (GPT-5 at 97.00%, Gemini 2.5 Pro at 96.19%, DeepSeek-R1 at 93.63%). With GRPO on synthetic data, Qwen3-8B-Base jumped from 23.18% to 70.94% (+47.76%), surpassing the reasoning-tuned Qwen3-8B-Thinking baseline (57.88%).
  • SSMR-Bench Visual QA: Qwen2.5-VL-7B-Instruct + GRPO reached 70.56% (up from 41.81%), outperforming Gemini 2.5 Pro (55.44%).
  • MusicTheoryBench (Real-World Benchmark): Qwen3-8B-Base + GRPO rose from 28.65% to 49.97%, beating zero-shot GPT-4 (41.90%) and nearing GPT-4 with Chain-of-Thought (52.55%).
  • Melodic Continuation Quality: On a 200-piece continuation task, Qwen3-8B-Base + GRPO achieved 76.00% Rhythmic Consistency (vs. 59.00% base) and 91.21% Chord Tone to Non-Chord Tone Ratio (CTnCTR) (vs. 88.03% base).

Limitations and Future Work

  • Foundational vs. Advanced Theory: Current templates focus on basic primitives (intervals, triads, meter, diatonic scales) and do not yet model complex progressions, cadences, modulation, or multi-phrase forms.
  • Visual Score Parsing: Sheet music image comprehension remains difficult across all tested architectures; better visual tokenizers for music scores are needed.
  • Future Directions: The authors plan to release an extensible Python toolkit leveraging music21 to support higher-level harmonic structures, alternative notations (e.g., guitar tablature, lead sheets), and aligned multi-modal representations (Symbolic–MIDI–Audio).

Summary updated

Abstract

Enhancing the ability of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) to interpret sheet music is a crucial step toward building AI musicians. However, current research lacks both evaluation benchmarks and training data for sheet music reasoning. Drawing from mathematics, where simple operations can generate a boundless universe of verifiable problems, we introduce a novel approach that treats core music theory rules, such as those governing beats and intervals, as programmatic functions to systematically synthesize a vast and diverse corpus of sheet music reasoning problems. This approach allows us to introduce a data synthesis framework that generates verifiable sheet music questions in both textual and visual modalities, leading to the Synthetic Sheet Music Reasoning Benchmark (SSMR-Bench) and a complementary training set. Evaluation results on SSMR-Bench highlight the key role reasoning plays in interpreting sheet music, while also pointing out the ongoing challenges in understanding sheet music in a visual format. By leveraging synthetic data for RLVR, models like Qwen3-8B-Base and Qwen2.5-VL-Instruct show significant improvements on the SSMR-Bench. Additionally, they also demonstrate considerable advancements on previously established human-crafted benchmarks, such as MusicTheoryBench and the music subset of MMMU. Finally, our results show that the enhanced reasoning ability can also facilitate music composition.
BibTeX
@misc{wang2025towards,
  author = {Zhilin Wang and Zhe Yang and Yun Luo and Yafu Li and Xiaoye Qu and Ziqian Qiao and Haoran Zhang and Runzhe Zhan and Derek F. Wong and Ji-Zhe Zhou and Yu Cheng},
  title = {{Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning}},
  year = {2025},
  eprint = {2509.04059},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.04059},
}
Figure