Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation

Ziya Zhou, Yuhang Wu, Zhiyue Wu, Xinyue Zhang, Ruibin Yuan, Yinghao Ma, Lu Wang, Emmanouil Benetos, Wei Xue, Yike Guo

Abstract

Symbolic Music, akin to language, can be encoded in discrete symbols. Recent research has extended the application of large language models (LLMs) such as GPT-4 and Llama2 to the symbolic music domain, including understanding and generation. Yet scant research explores the details of how these LLMs perform on advanced music understanding and conditioned generation, especially from the multi-step reasoning perspective, which is a critical aspect in the conditioned, editable, and interactive human-computer co-creation process. This study conducts a thorough investigation of LLMs' capability and limitations in symbolic music processing. We identify that current LLMs exhibit poor performance in song-level multi-step music reasoning and typically fail to leverage learned music knowledge when addressing complex musical tasks. Our findings suggest achieving advanced musical capability is not intrinsically obtained by LLMs, and future research should focus more on bridging the gap between music knowledge and reasoning to improve the co-creation experience for musicians.

Introduction

While Large Language Models (LLMs) have mastered the nuances of human language, their ability to "reason" within the musical domain remains a significant research frontier. Symbolic music—represented as text-based notations like ABC or MIDI—shares structural similarities with natural language, prompting researchers to apply models like GPT-4 and Llama2 to music tasks. However, music is not merely a linear sequence; it is a complex, multi-layered structure requiring hierarchical reasoning. This study investigates whether LLMs are truly reasoning through musical problems or simply predicting tokens based on surface-level patterns.

Methodology

The researchers evaluated four prominent models: GPT-4, Gemma-7B-it, Llama2-7B-chat, and Qwen-7B-chat. The evaluation spanned two primary categories:

  1. Music Understanding: Tasks included music theory exercises, motif extraction (identifying recurring melodic fragments), and musical form extraction (identifying the structural layout of a piece).
  2. Music Generation: Tasks included chord-conditioned generation, melody harmonization, and complex generation conditioned on both form and motif (MFMC).

To probe the models' reasoning depth, three prompting strategies were employed: Default (direct instruction), Chain-of-Thought (CoT) (step-by-step explanation), and In-Context Learning (ICL) (few-shot examples). Performance was measured using quantitative metrics (accuracy and ABC notation parsing rates) and qualitative human assessment by music experts.

Data Sources

The study utilized data from MusicPile and MusicBench (part of the ChatMusician project), covering six distinct tasks:

  • Music theory exercises: 367 items.
  • Motif extraction (ME): 2,470 items.
  • Musical form extraction (MFE): 483 items.
  • Chord-conditioned generation (CCG): 1,721 items.
  • Melody harmonization (MH): 355 items.
  • Musical-form-and-motif-conditioned generation (MFMC): 4,881 items.

Key Insights: The Knowledge-Reasoning Gap

A central finding of the study is the Knowledge Generalization Gap. LLMs often possess foundational musical knowledge—for instance, they can define a specific musical interval—yet they fail to apply that knowledge when analyzing ABC notation. This suggests that while the models have "memorized" music theory, they cannot reliably use it as a tool for reasoning.

Furthermore, the study identified a "Reversal Curse" in music. Similar to findings in logical reasoning tasks, LLMs may understand a musical concept in one direction but fail to apply it when the context is slightly altered or reversed.

In terms of creativity, the models often exhibited "Copy-Paste" behavior. When tasked with generating music based on a specific motif, models frequently repeated the motif verbatim rather than developing it into a cohesive melody. They struggled particularly with structural constraints, such as maintaining a consistent "AB" (verse-chorus) form.

Results and Analysis

  • Parsing and Syntax: GPT-4 was the most robust model, successfully producing renderable ABC notation approximately 79-80% of the time. Other models struggled significantly with syntax, with success rates often falling below 50%.
  • Reasoning vs. Knowledge: GPT-4 achieved nearly 70% accuracy on basic knowledge tasks using ICL, but this plummeted to ~35% on tasks requiring multi-step reasoning.
  • The CoT Paradox: Unlike in mathematics or logic, Chain-of-Thought (CoT) prompting provided limited benefits and occasionally hindered performance in smaller models. This often resulted in "hallucinations," where the model's step-by-step logic was fundamentally flawed despite following the correct format.
  • Model Comparison: Interestingly, the Gemma-7B-it model performed comparably to GPT-4 in specific music theory subsets, suggesting that smaller, specialized models may hold significant potential if properly tuned.

Conclusion and Future Work

The study concludes that advanced musical capability is not an emergent property of standard LLM training. Current models struggle with long-context symbolic music and fail to bridge the gap between theoretical knowledge and practical application.

Future research should move away from "one-shot" full-song generation, which remains unrealistic for current architectures. Instead, the authors suggest focusing on interactive, multi-round Supervised Fine-Tuning (SFT) that mimics the iterative workflow of human composers. Bridging the gap between music theory and symbolic reasoning will be essential for creating truly collaborative AI tools for musicians.

Summary updated

Abstract

Symbolic Music, akin to language, can be encoded in discrete symbols. Recent research has extended the application of large language models (LLMs) such as GPT-4 and Llama2 to the symbolic music domain including understanding and generation. Yet scant research explores the details of how these LLMs perform on advanced music understanding and conditioned generation, especially from the multi-step reasoning perspective, which is a critical aspect in the conditioned, editable, and interactive human-computer co-creation process. This study conducts a thorough investigation of LLMs' capability and limitations in symbolic music processing. We identify that current LLMs exhibit poor performance in song-level multi-step music reasoning, and typically fail to leverage learned music knowledge when addressing complex musical tasks. An analysis of LLMs' responses highlights distinctly their pros and cons. Our findings suggest achieving advanced musical capability is not intrinsically obtained by LLMs, and future research should focus more on bridging the gap between music knowledge and reasoning, to improve the co-creation experience for musicians.
BibTeX
@inproceedings{zhou2024can,
  author = {Ziya Zhou and Yuhang Wu and Zhiyue Wu and Xinyue Zhang and Ruibin Yuan and Yinghao Ma and Lu Wang and Emmanouil Benetos and Wei Xue and Yike Guo},
  title = {{Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation}},
  booktitle = {ISMIR},
  year = {2024},
  url = {https://openreview.net/forum?id=qZF5XjAX0V},
}
Figure