TL;DR
The Academic Text-to-Music (ATTM) Grand Challenge establishes a "fair-play" benchmark to democratize text-to-music generation, shifting the focus away from industrial data scale toward algorithmic efficiency. By restricting participants to a standardized, open-source, 3,777-hour instrumental dataset trained completely from scratch, the challenge provides a level playing field for academic research. It also introduces the Concept Coverage Score (CCS), a novel evaluation metric that leverages Large Audio Language Models (LALMs) to verify fine-grained semantic alignment in generated audio.
What Problem Does This Paper Address?
State-of-the-art text-to-music (TTM) systems are currently trapped behind a "compute and data wall." The field is dominated by commercial models trained on massive, proprietary datasets using industrial-scale hardware. This creates a massive barrier for academic researchers, who are often reduced to fine-tuning existing models or running small-scale experiments.
Without standardized training conditions, it is nearly impossible to ensure transparency, reproducibility, and true architectural innovation. The ATTM Grand Challenge addresses this gap by creating a strict framework where computational efficiency and musical intelligence take center stage over sheer data volume.
Methods
The challenge is split into two distinct tracks:
- Efficiency Track: Limits the core generative model to 500 million parameters, making it ideal for resource-constrained environments or edge-AI research.
- Performance Track: Imposes no parameter limits, challenging researchers to maximize musical quality within the data constraints.
Strict Constraints
To prevent "data laundering," participants must train their core generative models entirely from scratch using only the provided dataset. No pre-trained weights or synthetic data from commercial engines (like Suno or Udio) are allowed. However, teams can use open-source, pre-trained auxiliary components like audio tokenizers (e.g., EnCodec) and vocoders.
Multi-Stage Evaluation
Submissions are ranked using a combination of objective and subjective metrics:
- Fréchet Audio Distance (FAD): Measures acoustic quality against real music embeddings.
- CLAP Score: Evaluates global text-audio semantic alignment.
- Concept Coverage Score (CCS): A novel metric introduced in this paper. Instead of relying on broad embedding similarities, CCS uses a state-of-the-art LALM (Qwen3-Omni) as a zero-shot judge. The model inspects the log-probabilities of "Yes" and "No" tokens to verify if specific target concepts (genre, instrument, mood) requested in the prompt are actually present in the generated audio.
The objective metrics are aggregated using a Borda count method to determine which finalists advance to the phase-2 subjective Mean Opinion Score (MOS) listening tests.
Datasets
The challenge utilizes a curated, CC-licensed subset of the MTG-Jamendo dataset, specifically the raw_30s subset containing 55,701 tracks.
- Vocal Separation: Because the challenge focuses strictly on instrumental music, participants use a standardized pipeline featuring the Mel-Band Roformer model to strip out vocal stems. This yields a full dataset of 3,777 hours (~240 GB) or a computationally cheaper 30-second cropped subset of 464 hours (~25 GB).
- Audio Captioning: To provide semantic diversity, the organizers supplied two distinct reference caption sets generated by different LALM pipelines:
- Evaluation Set: A curated set of 100 test prompts divided into 80 In-Distribution (ID) triplets (tag combinations seen in the training data) and 20 Out-of-Distribution (OOD) triplets (unseen combinations like "heavy metal" & "calm") to test compositional generalizability.
Key Insights and Ideas
- Algorithmic Innovation Over Data Scale: By capping the data and prohibiting external pre-training, the challenge forced teams to innovate structurally. Successful teams explored diverse architectures, including Diffusion/Flow-Matching models, Transformers, and State Space Models (SSMs) like Mamba.
- LALMs as Granular Judges: The introduction of CCS proves that large audio-language models can serve as reliable, interpretable, zero-shot classifiers for audio evaluation, moving the industry closer to automated, fine-grained quality control.
- Lowering the Academic Barrier: Providing a 30-second cropped dataset subset allowed teams with limited hardware to remain highly competitive, proving that massive compute isn't always a prerequisite for meaningful AI music research.
Key Results
Out of 18 registered teams, 12 successfully submitted systems. Crucially, every single submission outperformed the official FluxAudio-S baseline.
- Efficiency Track Winner: Submission
e07took first place in the objective ranking, achieving the best FAD (0.417) and an exceptional CCS (0.867) using a Diffusion/Flow-Matching architecture. - Performance Track Winner: Submission
p05utilized a 2.4B parameter model and achieved the highest subjective MOS score among all participants (3.344).
When compared against industrial toplines, the best academic models approached but did not quite beat MusicGen-small (MOS 3.538). This makes sense, given that MusicGen-small was trained on 20,000 hours of licensed music—more than five times the size of the ATTM dataset. However, the narrow gap demonstrates how rapidly academic architectures are closing the performance divide under constrained conditions.
Limitations and Future Work
While the organizers successfully curated Out-of-Distribution (OOD) prompts to test how well models handle weird or unseen tag combinations, these scores were excluded from the final official leaderboard ranking. The authors noted that their objective evaluation metrics have not yet been fully verified on OOD samples, meaning they cannot yet produce completely reliable scores for official rankings. Validating and expanding automated metrics to handle highly compositional, out-of-distribution prompts remains an open area for future work.
Summary updated