CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following

Yinghao Ma, Siyou Li, Juntao Yu, Emmanouil Benetos, Akira Maezawa

TL;DR

CMI-Bench is a new, rigorous benchmark designed to evaluate how well audio-text Large Language Models (LLMs) follow instructions across 14 diverse Music Information Retrieval (MIR) tasks. The results reveal a sobering reality: while current LLMs are becoming "musically chatty," they still significantly underperform compared to specialized supervised models and exhibit notable cultural and gender biases.

What Problem Does This Paper Address?

The rise of audio-text LLMs (like Qwen2-Audio or SALMONN) has promised a future where we can interact with music through natural language. However, evaluating these models has been a "wild west." Most existing benchmarks rely on simplified Multiple-Choice Questions (MCQs) or basic captioning metrics that don't capture the technical complexity of music analysis.

Traditional Music Information Retrieval (MIR)—tasks like beat tracking, melody extraction, or precise instrument classification—requires a level of structural and temporal precision that "chatty" models often lack. There has been a significant gap between the general-purpose capabilities of LLMs and the rigorous, metric-driven standards of the MIR research community. CMI-Bench aims to bridge this gap by reinterpreting classic MIR tasks as instruction-following challenges.

Methods

The researchers developed a framework that transforms 14 core MIR tasks into natural language prompts. Instead of just picking A, B, or C, the models must generate open-ended responses that are then parsed and evaluated using standard MIR metrics.

Key methodological highlights include:

  • Instruction Reformulation: Tasks like "beat tracking" are turned into prompts asking for a comma-separated list of timestamps. "Key detection" asks the model to choose from 24 major/minor keys.
  • Standardized Evaluation: The team built a toolkit that maps LLM text outputs back into formats compatible with mir_eval, the industry-standard Python library for music analysis.
  • Semantic Tag Matching: For multi-label tasks (like music tagging), where a model might say "fiddle" instead of "violin," the researchers used the BGE encoder to calculate cosine similarity between predicted and ground-truth tags, allowing for a "soft" but semantically accurate evaluation.
  • Model Benchmarking: They evaluated 11 prominent open-source audio-text LLMs to establish a baseline of current capabilities.

Datasets

CMI-Bench is comprehensive, utilizing 20 diverse datasets across 14 tasks. These include:

  • Classification & Tagging: GTZAN (genre), MagnaTagATune (tagging), and MTG-Instrument.
  • Regression: EMO (arousal/valence).
  • Sequential/Temporal: GTZAN-Rhythm (beat tracking), MedleyDB (melody extraction), and DSing (lyrics transcription).
  • Specialized: GuZheng_99 (traditional Chinese instrument techniques).
  • Captioning: MusicCaps and SDD.

Key Insights and Ideas

  • The "Specialist" Gap: The most striking insight is that LLMs are currently "jacks of all trades, masters of none." Even the best-performing LLMs fall significantly short of task-specific supervised models. They lack the "inductive bias" that comes from being trained specifically for one musical task.
  • Data Contamination & Skew: Peak performance on specific tasks usually occurs when the evaluation dataset overlaps with the model's training corpus. This suggests that current models struggle to generalize to truly "unseen" musical distributions.
  • The Fragility of Instructions: Models are highly sensitive to how a prompt is phrased. Without very specific "task tokens" or clear schemas, their ability to interpret instructions can break down, especially for complex outputs like MIDI numbers or timestamps.
  • Perceptual Blindness: All tested models failed miserably at "Emotion Regression" (predicting how "energetic" or "calm" a song is on a scale). Their outputs often clustered around meaningless average values, showing a fundamental limitation in mapping continuous human perception to audio features.

Key Results

  • Top Performers: Qwen2-Audio and GAMA emerged as the strongest overall models, particularly in genre classification and tagging.
  • The Beat Tracking Failure: In tasks like beat tracking, models often simply "hallucinated" or copied example timestamps from the prompt rather than actually analyzing the audio. For example, Qwen2-Audio's performance on the Ballroom dataset was largely due to it repeating the input examples.
  • Lyrics Transcription: Despite having powerful speech encoders (like Whisper), most models performed poorly on the DSing dataset, highlighting that singing voice transcription remains much harder for LLMs than standard speech-to-text.
  • Bias: The evaluation uncovered a persistent Western bias. Models performed well on piano and violin but saw significant performance drops on instruments like the bongo or harmonica. There was also a gender bias; some models were significantly better at identifying female voices than male ones.

Limitations and Future Work

The authors acknowledge several hurdles:

  1. Cultural Narrowness: The benchmark is still heavily weighted toward Western, English-language popular music.
  2. Temporal Precision: LLMs currently struggle with "dense temporal supervision"—knowing exactly when something happens in a song.
  3. Continuous Attributes: Mapping audio to a numerical scale (like 1-9 for arousal) remains an unsolved problem for these architectures.

Future Work: The researchers point toward the need for better "timestamped" pre-training data and more culturally inclusive datasets to mitigate the biases found in current foundation models. They hope CMI-Bench will serve as a standardized "North Star" for the development of more music-aware AI.

Summary updated

Abstract

Recent advances in audio-text large language models (LLMs) have opened new possibilities for music understanding and generation. However, existing benchmarks are limited in scope, often relying on simplified tasks or multi-choice evaluations that fail to reflect the complexity of real-world music analysis. We reinterpret a broad range of traditional MIR annotations as instruction-following formats and introduce CMI-Bench, a comprehensive music instruction following benchmark designed to evaluate audio-text LLMs on a diverse set of music information retrieval (MIR) tasks. These include genre classification, emotion regression, emotion tagging, instrument classification, pitch estimation, key detection, lyrics transcription, melody extraction, vocal technique recognition, instrument performance technique detection, music tagging, music captioning, and (down)beat tracking: reflecting core challenges in MIR research. Unlike previous benchmarks, CMI-Bench adopts standardized evaluation metrics consistent with previous state-of-the-art MIR models, ensuring direct comparability with supervised approaches. We provide an evaluation toolkit supporting all open-source audio-textual LLMs, including LTU, Qwen-audio, SALMONN, MusiLingo, etc. Experiment results reveal significant performance gaps between LLMs and supervised models, along with their culture, chronological and gender bias, highlighting the potential and limitations of current models in addressing MIR tasks. CMI-Bench establishes a unified foundation for evaluating music instruction following, driving progress in music-aware LLMs.
BibTeX
@inproceedings{ma2025cmi,
  author = {Yinghao Ma and Siyou Li and Juntao Yu and Emmanouil Benetos and Akira Maezawa},
  title = {{CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following}},
  booktitle = {ISMIR},
  year = {2025},
  url = {https://openreview.net/forum?id=2amYHY66Aw},
}

Loading related content...

Figure