Evaluating how well an AI generates audio from a text prompt has traditionally been a "black box" affair. You either pay humans to listen and rate clips—which is slow and expensive—or you use automated metrics like CLAPScore that provide a single, often opaque number with mediocre correlation to human judgment.
A new paper from researchers at National Taiwan University and Amazon AGI introduces CoTTAGE (Chain-of-Thought Text-to-Audio Generation Evaluator), a reasoning-based judge that doesn't just give a score; it explains its "thought process" in plain English.
TL;DR
CoTTAGE is an explainable evaluator for text-to-audio generation that follows a "reason-then-predict" paradigm to provide both accurate scores and audio-grounded rationales. By leveraging a novel Thurstone-based reinforcement learning objective and over 10k new human annotations, it achieves state-of-the-art correlation with human judgment while producing explanations that humans actually prefer over those from frontier models.
What Problem Does This Paper Address?
The rapid advancement of Text-to-Audio (TTA) models has outpaced our ability to evaluate them. Current automatic metrics primarily rely on embedding similarities (like CLAP), which fail to capture nuanced temporal misalignments or subtle quality artifacts. Furthermore, these metrics offer no diagnostic feedback. If a model gets a low score, the developer doesn't know why.
While Large Audio Language Models (LALMs) like Gemini or Qwen-Omni can technically "talk" about audio, they often suffer from hallucinations—describing sounds that aren't there or simply repeating the text prompt instead of listening to the audio. This creates a bottleneck where researchers must still rely on labor-intensive human evaluation for reliable results.
Methods
CoTTAGE is built on the Qwen2.5-Omni-7B backbone and trained in a sophisticated two-stage pipeline:
- Supervised Fine-Tuning (SFT): The model is first trained on "reasoning traces." The authors used Gemini-3.1 Pro to generate rationales for existing datasets, conditioned on the ground-truth human scores. This teaches CoTTAGE the structure of a good evaluation: analyze the audio evidence first, then conclude with a score.
- Reinforcement Learning (RL): To further align the model with human preferences, the authors employed Grouped Relative Policy Optimization (GRPO). They introduced a novel Thurstone-based pairwise preference reward. Instead of just regressing to a point estimate, the model generates multiple reasoning trajectories. The reward is calculated by converting the resulting score distribution (mean and variance) into pairwise win probabilities, effectively rewarding the model for being both accurate and certain.
Datasets
The researchers utilized several key benchmarks:
- RELATE & XACLE: Originally designed for Text Relevance (REL), the authors contributed 10,362 new human annotations for Overall Quality (OVL) to these sets to enable joint training.
- PAM-Audio & Clotho: Used as out-of-domain test sets to prove the model's ability to generalize beyond its training data.
Key Insights and Ideas
- Reasoning Improves Accuracy: Forcing the model to generate a rationale before the score (Chain-of-Thought) isn't just for show—it significantly boosts the correlation with human judgments.
- General LALMs are Poor Judges: Out of the box, even powerful models like Qwen3-Omni-30B show poor correlation with human quality ratings. Audio understanding is a necessary but insufficient condition for audio evaluation; task-specific alignment is critical.
- Aesthetics $\neq$ Quality: The authors found that existing "aesthetic" metrics (like production quality or enjoyment) are poor proxies for the "Overall Quality" (OVL) humans care about in TTA. This justifies the need for their newly collected 10k+ OVL labels.
- Uncertainty-Aware RL: By using a Thurstone model in the RL stage, the system naturally yields "softer" rewards when the model is inconsistent across different reasoning paths, leading to more stable and conservative training.
Key Results
CoTTAGE consistently outperformed all baselines, including proprietary frontier models, across six different benchmarks:
- Superior Correlation: On the XACLE benchmark for text relevance, CoTTAGE achieved a Pearson Correlation (LCC) of 0.685, dwarfing CLAPScore (0.472) and Gemini-3.1 Pro (0.551).
- Human-Preferred Explanations: In blinded head-to-head tests, human annotators preferred CoTTAGE's reasoning over Gemini-3.1 Pro's in 65.3% of cases for relevance and 53.4% for quality.
- RL Boost: The reinforcement learning stage provided a consistent gain of roughly 0.05 to 0.08 in correlation metrics over the SFT-only version, proving the value of the pairwise preference objective.
Limitations and Future Work
Despite its success, the authors note a few areas for growth:
- Short-form Focus: The current benchmarks and model are optimized for clips under 30 seconds. Evaluating long-form "soundscapes" remains a challenge.
- Lack of Precise Timestamps: While CoTTAGE describes what is wrong (e.g., "the snoring starts too early"), it doesn't yet output exact millisecond timestamps for errors.
- Domain Specificity: The training data is largely environmental sounds. Extending this to music-centric or speech-centric generation is a promising next step.
CoTTAGE represents a significant step toward "Glass Box" AI evaluation, providing TTA developers with a tool that is not only more accurate but also helps them understand exactly where their models are falling silent.
Summary updated