Abstract
Recent advances in audio-text large language models (LLMs) have opened new possibilities for music understanding and generation. However, existing benchmarks are limited in scope, often relying on simplified tasks or multi-choice evaluations that fail to reflect the complexity of real-world music analysis. We reinterpret a broad range of traditional MIR annotations as instruction-following formats and introduce CMI-Bench, a comprehensive music instruction following benchmark designed to evaluate audio-text LLMs on a diverse set of music information retrieval (MIR) tasks. These include genre classification, emotion regression, emotion tagging, instrument classification, pitch estimation, key detection, lyrics transcription, melody extraction, vocal technique recognition, instrument performance technique detection, music tagging, music captioning, and (down)beat tracking: reflecting core challenges in MIR research. Unlike previous benchmarks, CMI-Bench adopts standardized evaluation metrics consistent with previous state-of-the-art MIR models, ensuring direct comparability with supervised approaches. We provide an evaluation toolkit supporting all open-source audio-textual LLMs, including LTU, Qwen-audio, SALMONN, MusiLingo, etc. Experiment results reveal significant performance gaps between LLMs and supervised models, along with their culture, chronological and gender bias, highlighting the potential and limitations of current models in addressing MIR tasks. CMI-Bench establishes a unified foundation for evaluating music instruction following, driving progress in music-aware LLMs.
Introduction
The rise of audio-text Large Language Models (LLMs), such as Qwen2-Audio and SALMONN, has promised a future where users can interact with music through natural language. However, the evaluation of these models has remained inconsistent. Most existing benchmarks rely on simplified Multiple-Choice Questions (MCQs) or basic captioning metrics that fail to capture the technical complexity of music analysis.
Traditional Music Information Retrieval (MIR)—encompassing tasks like beat tracking, melody extraction, and precise instrument classification—requires a level of structural and temporal precision that general-purpose "chatty" models often lack. CMI-Bench aims to bridge the gap between the general capabilities of LLMs and the rigorous, metric-driven standards of the MIR research community by reinterpreting classic MIR tasks as instruction-following challenges.
Methodology
The CMI-Bench framework transforms 14 core MIR tasks into natural language prompts. Instead of selecting from fixed options, models must generate open-ended responses that are subsequently parsed and evaluated using standard MIR metrics.
Instruction Reformulation and Evaluation
Key methodological innovations include:
- Task Reformulation: Complex tasks are converted into specific instructions. For instance, beat tracking prompts ask for a comma-separated list of timestamps, while key detection requires the model to identify one of 24 major or minor keys.
- Standardized Evaluation Toolkit: The researchers developed a toolkit to map LLM text outputs back into formats compatible with
mir_eval, the industry-standard Python library for music analysis. This ensures that LLM performance can be directly compared to traditional supervised models. - Semantic Tag Matching: To handle multi-label tasks where models might use synonyms (e.g., "fiddle" instead of "violin"), the framework uses the BGE encoder to calculate cosine similarity between predicted and ground-truth tags, allowing for semantically accurate "soft" evaluation.
Benchmark Composition
CMI-Bench is a comprehensive suite utilizing 20 diverse datasets across 14 tasks, categorized as follows:
- Classification & Tagging: GTZAN (genre), MagnaTagATune (tagging), and MTG-Instrument.
- Regression: EMO (arousal and valence).
- Sequential & Temporal: GTZAN-Rhythm (beat tracking), MedleyDB (melody extraction), and DSing (lyrics transcription).
- Specialized MIR: GuZheng_99 (traditional Chinese instrument techniques).
- Captioning: MusicCaps and SDD.
Key Insights and Results
The evaluation of 11 prominent open-source audio-text LLMs revealed a sobering reality regarding the current state of music-aware AI.
The Specialist Gap
The most striking finding is that current LLMs are "jacks of all trades, masters of none." Even the highest-performing LLMs fall significantly short of task-specific supervised models. They lack the inductive bias that comes from specialized training on musical structures.
Temporal and Perceptual Limitations
Models struggled immensely with tasks requiring high temporal precision. In beat tracking, models often "hallucinated" or simply echoed example timestamps provided in the prompt. Furthermore, all tested models failed in Emotion Regression (predicting arousal/valence on a numerical scale). Their outputs typically clustered around meaningless average values, indicating a fundamental limitation in mapping continuous human perception to audio features.
Performance Leaders and Failures
- Top Performers: Qwen2-Audio and GAMA emerged as the strongest overall models, particularly in genre classification and tagging.
- Lyrics Transcription: Despite utilizing powerful speech encoders like Whisper, most models performed poorly on the DSing dataset, highlighting that singing voice transcription remains significantly more difficult for LLMs than standard speech-to-text.
Bias and Generalization
The evaluation uncovered persistent Western bias; models performed well on piano and violin but saw significant performance drops on instruments like the bongo or harmonica. Additionally, a gender bias was observed, with some models identifying female voices more accurately than male ones. Performance also peaked when evaluation data overlapped with the model's training corpus, suggesting a struggle to generalize to truly unseen musical distributions.
Conclusion
CMI-Bench establishes a standardized "North Star" for the development of music-aware LLMs. While these models show promise in general music interaction, they currently lack the precision and cultural breadth required for professional MIR tasks. Future work must focus on improving temporal supervision, addressing continuous attribute mapping, and expanding datasets to include more culturally diverse musical traditions.
Summary updated