Academic Text-to-Music Grand Challenge: Datasets, Baselines, and Evaluation Methods

Fang-Chih Hsieh, Wei-Jaw Lee, Chun-Ping Wang, Hung-yi Lee, Hao-Wen Dong, Yi-Hsuan YangarXiv 2026 · arXiv

TL;DR

The Academic Text-to-Music (ATTM) Grand Challenge establishes a "fair-play" benchmark to democratize text-to-music generation, shifting the focus away from industrial data scale toward algorithmic efficiency. By restricting participants to a standardized, open-source, 3,777-hour instrumental dataset trained completely from scratch, the challenge provides a level playing field for academic research. It also introduces the Concept Coverage Score (CCS), a novel evaluation metric that leverages Large Audio Language Models (LALMs) to verify fine-grained semantic alignment in generated audio.

What Problem Does This Paper Address?

State-of-the-art text-to-music (TTM) systems are currently trapped behind a "compute and data wall." The field is dominated by commercial models trained on massive, proprietary datasets using industrial-scale hardware. This creates a massive barrier for academic researchers, who are often reduced to fine-tuning existing models or running small-scale experiments.

Without standardized training conditions, it is nearly impossible to ensure transparency, reproducibility, and true architectural innovation. The ATTM Grand Challenge addresses this gap by creating a strict framework where computational efficiency and musical intelligence take center stage over sheer data volume.

Methods

The challenge is split into two distinct tracks:

  1. Efficiency Track: Limits the core generative model to 500 million parameters, making it ideal for resource-constrained environments or edge-AI research.
  2. Performance Track: Imposes no parameter limits, challenging researchers to maximize musical quality within the data constraints.

Strict Constraints

To prevent "data laundering," participants must train their core generative models entirely from scratch using only the provided dataset. No pre-trained weights or synthetic data from commercial engines (like Suno or Udio) are allowed. However, teams can use open-source, pre-trained auxiliary components like audio tokenizers (e.g., EnCodec) and vocoders.

Multi-Stage Evaluation

Submissions are ranked using a combination of objective and subjective metrics:

  • Fréchet Audio Distance (FAD): Measures acoustic quality against real music embeddings.
  • CLAP Score: Evaluates global text-audio semantic alignment.
  • Concept Coverage Score (CCS): A novel metric introduced in this paper. Instead of relying on broad embedding similarities, CCS uses a state-of-the-art LALM (Qwen3-Omni) as a zero-shot judge. The model inspects the log-probabilities of "Yes" and "No" tokens to verify if specific target concepts (genre, instrument, mood) requested in the prompt are actually present in the generated audio.

The objective metrics are aggregated using a Borda count method to determine which finalists advance to the phase-2 subjective Mean Opinion Score (MOS) listening tests.

Datasets

The challenge utilizes a curated, CC-licensed subset of the MTG-Jamendo dataset, specifically the raw_30s subset containing 55,701 tracks.

  • Vocal Separation: Because the challenge focuses strictly on instrumental music, participants use a standardized pipeline featuring the Mel-Band Roformer model to strip out vocal stems. This yields a full dataset of 3,777 hours (~240 GB) or a computationally cheaper 30-second cropped subset of 464 hours (~25 GB).
  • Audio Captioning: To provide semantic diversity, the organizers supplied two distinct reference caption sets generated by different LALM pipelines:
Prompts used for generating the reference captions
  • Evaluation Set: A curated set of 100 test prompts divided into 80 In-Distribution (ID) triplets (tag combinations seen in the training data) and 20 Out-of-Distribution (OOD) triplets (unseen combinations like "heavy metal" & "calm") to test compositional generalizability.

Key Insights and Ideas

  • Algorithmic Innovation Over Data Scale: By capping the data and prohibiting external pre-training, the challenge forced teams to innovate structurally. Successful teams explored diverse architectures, including Diffusion/Flow-Matching models, Transformers, and State Space Models (SSMs) like Mamba.
  • LALMs as Granular Judges: The introduction of CCS proves that large audio-language models can serve as reliable, interpretable, zero-shot classifiers for audio evaluation, moving the industry closer to automated, fine-grained quality control.
  • Lowering the Academic Barrier: Providing a 30-second cropped dataset subset allowed teams with limited hardware to remain highly competitive, proving that massive compute isn't always a prerequisite for meaningful AI music research.

Key Results

Out of 18 registered teams, 12 successfully submitted systems. Crucially, every single submission outperformed the official FluxAudio-S baseline.

Comprehensive benchmarking summary for objective evaluation
  • Efficiency Track Winner: Submission e07 took first place in the objective ranking, achieving the best FAD (0.417) and an exceptional CCS (0.867) using a Diffusion/Flow-Matching architecture.
  • Performance Track Winner: Submission p05 utilized a 2.4B parameter model and achieved the highest subjective MOS score among all participants (3.344).
Subjective evaluation results based on overall MOS scores

When compared against industrial toplines, the best academic models approached but did not quite beat MusicGen-small (MOS 3.538). This makes sense, given that MusicGen-small was trained on 20,000 hours of licensed music—more than five times the size of the ATTM dataset. However, the narrow gap demonstrates how rapidly academic architectures are closing the performance divide under constrained conditions.

Limitations and Future Work

While the organizers successfully curated Out-of-Distribution (OOD) prompts to test how well models handle weird or unseen tag combinations, these scores were excluded from the final official leaderboard ranking. The authors noted that their objective evaluation metrics have not yet been fully verified on OOD samples, meaning they cannot yet produce completely reliable scores for official rankings. Validating and expanding automated metrics to handle highly compositional, out-of-distribution prompts remains an open area for future work.

Summary updated

Abstract

This paper presents an overview and the technical framework of the ICME 2026 Grand Challenge on Academic Text-to-Music Generation (ATTM). Despite the rapid progress in text-to-music generation (TTM) systems, the field is currently dominated by models trained on massive proprietary datasets with industrial-scale computational resources, creating a significant barrier for academic research. To address this, the ATTM Challenge establishes a fair-play benchmark that requires participants to train generative models strictly from scratch using a standardized, CC-licensed subset of the MTG-Jamendo dataset containing only instrumental music. The challenge is divided into two tracks: the Efficiency Track (limited to 500M parameters) and the Performance Track (no parameter limit). Submissions are evaluated through a multi-stage process involving objective metrics, including Frechet Audio Distance, CLAP score, and a novel Concept Coverage Score (CCS), followed by a subjective listening test. By providing open-source baselines, preprocessing pipelines, reference captions, and public evaluation code for computing FAD and CLAP, this challenge aims to facilitate and promote TTM research in academic contexts.
BibTeX
@misc{hsieh2026academic,
  author = {Fang-Chih Hsieh and Wei-Jaw Lee and Chun-Ping Wang and Hung-yi Lee and Hao-Wen Dong and Yi-Hsuan Yang},
  title = {{Academic Text-to-Music Grand Challenge: Datasets, Baselines, and Evaluation Methods}},
  year = {2026},
  eprint = {2605.21538},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.21538},
}
Figure