TL;DR
CMI-Bench is a new, rigorous benchmark designed to evaluate how well audio-text Large Language Models (LLMs) follow instructions across 14 diverse Music Information Retrieval (MIR) tasks. The results reveal a sobering reality: while current LLMs are becoming "musically chatty," they still significantly underperform compared to specialized supervised models and exhibit notable cultural and gender biases.
What Problem Does This Paper Address?
The rise of audio-text LLMs (like Qwen2-Audio or SALMONN) has promised a future where we can interact with music through natural language. However, evaluating these models has been a "wild west." Most existing benchmarks rely on simplified Multiple-Choice Questions (MCQs) or basic captioning metrics that don't capture the technical complexity of music analysis.
Traditional Music Information Retrieval (MIR)—tasks like beat tracking, melody extraction, or precise instrument classification—requires a level of structural and temporal precision that "chatty" models often lack. There has been a significant gap between the general-purpose capabilities of LLMs and the rigorous, metric-driven standards of the MIR research community. CMI-Bench aims to bridge this gap by reinterpreting classic MIR tasks as instruction-following challenges.
Methods
The researchers developed a framework that transforms 14 core MIR tasks into natural language prompts. Instead of just picking A, B, or C, the models must generate open-ended responses that are then parsed and evaluated using standard MIR metrics.
Key methodological highlights include:
- Instruction Reformulation: Tasks like "beat tracking" are turned into prompts asking for a comma-separated list of timestamps. "Key detection" asks the model to choose from 24 major/minor keys.
- Standardized Evaluation: The team built a toolkit that maps LLM text outputs back into formats compatible with
mir_eval, the industry-standard Python library for music analysis. - Semantic Tag Matching: For multi-label tasks (like music tagging), where a model might say "fiddle" instead of "violin," the researchers used the BGE encoder to calculate cosine similarity between predicted and ground-truth tags, allowing for a "soft" but semantically accurate evaluation.
- Model Benchmarking: They evaluated 11 prominent open-source audio-text LLMs to establish a baseline of current capabilities.
Datasets
CMI-Bench is comprehensive, utilizing 20 diverse datasets across 14 tasks. These include:
- Classification & Tagging: GTZAN (genre), MagnaTagATune (tagging), and MTG-Instrument.
- Regression: EMO (arousal/valence).
- Sequential/Temporal: GTZAN-Rhythm (beat tracking), MedleyDB (melody extraction), and DSing (lyrics transcription).
- Specialized: GuZheng_99 (traditional Chinese instrument techniques).
- Captioning: MusicCaps and SDD.
Key Insights and Ideas
- The "Specialist" Gap: The most striking insight is that LLMs are currently "jacks of all trades, masters of none." Even the best-performing LLMs fall significantly short of task-specific supervised models. They lack the "inductive bias" that comes from being trained specifically for one musical task.
- Data Contamination & Skew: Peak performance on specific tasks usually occurs when the evaluation dataset overlaps with the model's training corpus. This suggests that current models struggle to generalize to truly "unseen" musical distributions.
- The Fragility of Instructions: Models are highly sensitive to how a prompt is phrased. Without very specific "task tokens" or clear schemas, their ability to interpret instructions can break down, especially for complex outputs like MIDI numbers or timestamps.
- Perceptual Blindness: All tested models failed miserably at "Emotion Regression" (predicting how "energetic" or "calm" a song is on a scale). Their outputs often clustered around meaningless average values, showing a fundamental limitation in mapping continuous human perception to audio features.
Key Results
- Top Performers: Qwen2-Audio and GAMA emerged as the strongest overall models, particularly in genre classification and tagging.
- The Beat Tracking Failure: In tasks like beat tracking, models often simply "hallucinated" or copied example timestamps from the prompt rather than actually analyzing the audio. For example, Qwen2-Audio's performance on the Ballroom dataset was largely due to it repeating the input examples.
- Lyrics Transcription: Despite having powerful speech encoders (like Whisper), most models performed poorly on the DSing dataset, highlighting that singing voice transcription remains much harder for LLMs than standard speech-to-text.
- Bias: The evaluation uncovered a persistent Western bias. Models performed well on piano and violin but saw significant performance drops on instruments like the bongo or harmonica. There was also a gender bias; some models were significantly better at identifying female voices than male ones.
Limitations and Future Work
The authors acknowledge several hurdles:
- Cultural Narrowness: The benchmark is still heavily weighted toward Western, English-language popular music.
- Temporal Precision: LLMs currently struggle with "dense temporal supervision"—knowing exactly when something happens in a song.
- Continuous Attributes: Mapping audio to a numerical scale (like 1-9 for arousal) remains an unsolved problem for these architectures.
Future Work: The researchers point toward the need for better "timestamped" pre-training data and more culturally inclusive datasets to mitigate the biases found in current foundation models. They hope CMI-Bench will serve as a standardized "North Star" for the development of more music-aware AI.
Summary updated