Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning

Zhilin Wang, Zhe Yang, Yun Luo, Yafu Li, Xiaoye Qu, Ziqian Qiao, Haoran Zhang, Runzhe Zhan, Derek F. Wong, Jizhe Zhou, Yu Cheng

Abstract

Enhancing the ability of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) to interpret sheet music is a crucial step toward building AI musicians. However, current research lacks both evaluation benchmarks and training data for sheet music reasoning. Inspired by mathematics, where simple operations yield infinite verifiable problems, we introduce a novel approach that treats core music theory rules, such as those governing beats and intervals, as programmatic functions to systematically synthesize a vast and diverse corpus of sheet music reasoning problems. This approach allows us to introduce a data synthesis framework that generates verifiable sheet music questions in both textual and visual modalities, leading to the Synthetic Sheet Music Reasoning Benchmark (SSMR-Bench) and a complementary training set. Evaluation results on SSMR-Bench highlight the key role reasoning plays in interpreting sheet music, while also pointing out the ongoing challenges in understanding sheet music in a visual format. By leveraging synthetic data for RLVR, all models show significant improvements on the SSMR-Bench. Additionally, they also demonstrate considerable advancements on previously established human-crafted benchmarks, such as MusicTheoryBench and the music subset of MMMU. Finally, our results show that the enhanced reasoning ability can also facilitate music composition.

Introduction

While Large Language Models (LLMs) have become proficient at music "knowledge" (e.g., identifying composers) and "recognition" (e.g., optical music recognition), they struggle with musical reasoning. This involves actively applying music theory rules to interpret a score—such as calculating the total beats in a complex measure or identifying a chord from a set of disparate notes. The primary bottleneck is data: human-crafted music theory exams are small and do not scale. Without a massive corpus of problems with "verifiable" answers, it is difficult to use advanced training techniques like Reinforcement Learning to teach models how to "think" through a musical score.

The SSMR Framework

We propose a "math-inspired" approach to music. Just as arithmetic operations can generate infinite math problems, music theory rules (beats, intervals, scales) can be treated as programmatic functions.

The programmatic data synthesis pipeline for generating textual and visual music reasoning problems.

Our framework uses ABC notation—a text-based format for music—as the foundation. We developed nine templates across four domains: Rhythm, Chord, Interval, and Scale. For example, a "Bar Placement" function takes a string of notes, removes the bar lines, and asks the model to re-insert them based on the time signature. These textual problems are then programmatically rendered into standard visual staff notation images, creating a parallel multimodal dataset.

Training with Verifiable Rewards

Using the Group Relative Policy Optimization (GRPO) algorithm, we trained models via Reinforcement Learning with Verifiable Rewards (RLVR). Because the problems are synthesized, the "reward" is simple: did the model get the right answer? This allows the model to "learn to reason" through trial and error. We evaluated this approach using SSMR-Bench, which consists of 1,600 textual and 1,600 visual QA pairs.

Key Results and Insights

The impact of RLVR training on synthetic data was profound. Qwen3-8B-Base saw its average score on the SSMR-Bench jump from 23.18% to 70.94%. Furthermore, on the human-crafted MusicTheoryBench, the synthetically trained model improved by over 21%, eventually outperforming GPT-4.

Performance improvements on math and music benchmarks after RLVR training on synthetic music data.

A particularly fascinating finding was the cross-domain synergy. Training on music reasoning improved the models' performance on mathematical benchmarks like AIME24 and AMC. This suggests that the logical structures required for music theory (counting beats, calculating intervals) share a fundamental reasoning backbone with mathematics.

However, a significant "reasoning gap" remains between modalities. Even top-tier models like Gemini 2.5 Pro score ~96% on text but drop to ~55% on images, suggesting that visual perception of music is currently a bigger hurdle than logical understanding.

Conclusion

By treating music theory as a set of verifiable programmatic functions, we have created a scalable path for teaching AI to reason about music. Our results demonstrate that synthetic data not only improves performance on music benchmarks but also enhances general reasoning and composition abilities. Future work will focus on closing the visual perception gap and expanding the framework to complex polyphonic textures and harmonic analysis.

Summary updated

Abstract

Enhancing the ability of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) to interpret sheet music is a crucial step toward building AI musicians. However, current research lacks both evaluation benchmarks and training data for sheet music reasoning. Inspired by mathematics, where simple operations yield infinite verifiable problems, we introduce a novel approach that treats core music theory rules, such as those governing beats and intervals, as programmatic functions to systematically synthesize a vast and diverse corpus of sheet music reasoning problems. This approach allows us to introduce a data synthesis framework that generates verifiable sheet music questions in both textual and visual modalities, leading to the Synthetic Sheet Music Reasoning Benchmark (SSMR-Bench) and a complementary training set. Evaluation results on SSMR-Bench highlight the key role reasoning plays in interpreting sheet music, while also pointing out the ongoing challenges in understanding sheet music in a visual format. By leveraging synthetic data for RLVR, all models show significant improvements on the SSMR-Bench. Additionally, they also demonstrate considerable advancements on previously established human-crafted benchmarks, such as MusicTheoryBench and the music subset of MMMU. Finally, our results show that the enhanced reasoning ability can also facilitate music composition.
BibTeX
@misc{wang2025towards,
  author = {Zhilin Wang and Zhe Yang and Yun Luo and Yafu Li and Xiaoye Qu and Ziqian Qiao and Haoran Zhang and Runzhe Zhan and Derek F. Wong and Jizhe Zhou and Yu Cheng},
  title = {{Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning}},
  year = {2025},
  eprint = {2509.04059},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.04059},
}
Figure