TL;DR
Can general-purpose large language models (LLMs) genuinely "reason" through musical structures, or are they merely regurgitating surface patterns? Evaluating four popular LLMs across symbolic music understanding and generation tasks reveals a striking gap: while models like GPT-4 can follow basic conversational directions, all evaluated LLMs struggle with song-level, multi-step musical reasoning and frequently misapply foundational music theory when working with musical notation.
What Problem Does This Paper Address?
Recent advances have seen LLMs applied to symbolic music (such as ABC notation and MIDI), treating musical tokens analogously to text. However, composing or analyzing music requires more than syntactic fluency—it demands music reasoning: deducing latent harmonic progressions, motifs, rhythm, and formal structures that are not explicitly tagged in the raw score.
While techniques like Chain-of-Thought (CoT) prompting substantially boost LLM performance in logical domains like mathematics, it remains unclear whether these capabilities translate to symbolic music. This paper systematically investigates how well modern LLMs reason through multi-step symbolic music tasks in both comprehension and conditioned generation.
Methods
The authors evaluated four prominent models: GPT-4, Gemma-7B-it, Llama2-7B-chat, and Qwen-7B-chat.
The investigation benchmarked performance across six distinct tasks:
- Music Understanding: Music theory exercises, motif extraction (ME), and musical form extraction (MFE).
- Conditioned Music Generation: Chord-conditioned generation (CCG), melody harmonization (MH), and musical-form-and-motif-conditioned generation (MFMC).
Key Methodological Elements:
- Prompt Engineering Modes: Models were tested using Default (direct output), Chain-of-Thought (CoT) (step-by-step reasoning), and In-Context Learning (ICL). For generative tasks, the authors designed a novel 4-round conversational prompt protocol formulated by surveying professional music composition graduates to mimic human compositional workflows.
- Format & Execution Pipeline: Extracted ABC notation was converted to MIDI via
Music21and synthesized to audio viamidi2audioto measure syntactic validity. - Dual-Track Evaluation: Combined quantitative parsing success and multi-choice accuracy with qualitative evaluations (blind A/B listening tests and multi-metric human evaluations by musical experts scoring instruction-following, human-like reasoning, and constraint satisfaction).
Datasets
The authors utilized data sampled from MusicPile and MusicBench (introduced in ChatMusician):
- Music Theory Exercises: 367 items split into basic Knowledge and Reasoning (computational) subsets.
- Motif Extraction (ME): 2,470 items.
- Musical Form Extraction (MFE): 483 items.
- Chord-Conditioned Generation (CCG): 1,721 items.
- Melody Harmonization (MH): 355 items.
- Form-and-Motif-Conditioned Generation (MFMC): 4,881 items.
Key Insights and Ideas
- The Musical "Reversal Curse": LLMs exhibit a severe knowledge-generalization gap. They can accurately state declarative definitions (e.g., defining note intervals or describing ABC syntax), yet they fail completely when applying these concepts to raw ABC text. For example, GPT-4 mistook the number
"4"in an ABC string—which designated note duration—as meaning notes were "a fourth apart," corrupting its subsequent interval calculations. - Standard Prompt Engineering Often Fails in Music: While CoT and ICL improved GPT-4's reasoning accuracy on music theory (from 25.6% to 36.7%), they offered minimal benefit—or even actively degraded performance—on the 7B open-source models (e.g., Gemma dropped from 31.6% to 17.3% under CoT).
- Superficial Adherence over Compositional Depth: When tasked with complex generative constraints, models frequently hallucinated by simply copying the prompt's motif, producing erratic harmonic repetitions, or introducing off-key pitches rather than integrating the motif into a coherent macro-structure (such as an "AB" form).
Key Results
- Syntactic Execution (Audio Renderability): Except for GPT-4, open-source 7B models frequently failed to output parseable ABC notation. GPT-4 achieved an audio generation success rate of ~79–80% on CCG/MH and 47% on MFMC. In contrast, Qwen-7B-chat and Gemma-7B-it achieved under 35%, while Llama2-7B-chat achieved 0% valid renders across all generative benchmarks.
- Music Theory Performance: In the Reasoning subset, GPT-4 scored 25.6% (Default) and 36.7% (CoT). Gemma-7B-it showed surprisingly strong baseline reasoning in Default mode (31.6%), outperforming Llama2 and Qwen, though it degraded when forced into step-by-step analysis.
- Human Expert Assessment (0–10 Scale): In MFE and ME tasks, GPT-4 was the only model capable of reliably following instructions (10.0/10), but its human-like reasoning score dropped to 5.6/10 on musical form extraction. On generative musicality, human evaluators found GPT-4 and Gemma comparable on MFMC, but noted that LLM compositions across the board were overly repetitive, rhythmically primitive, and significantly behind human-composed baselines.
Limitations and Future Work
- Context Window Mismatch: Despite claiming 4K–8K token context capabilities, none of the LLMs successfully leveraged long-context representations to capture structural relationships in symbolic music.
- Flawed Baseline Benchmarks: Human composers noted that existing evaluation datasets contain unrealistic conditions (such as unmusical bass line progressions in chord sequences), which confound LLM evaluation.
- Path Forward: The authors advocate moving away from one-step generation. Instead, future progress requires constructing specialized Supervised Fine-Tuning (SFT) datasets structured around expert compositional workflows—explicitly decomposing generation into motif drafting, harmonic planning, self-evaluation, and iterative refinement.
Summary updated