CoTTAGE: An Explainable Judge for Text-to-Audio Generation

Yu-Xiang Lin, Yun-Man Hsu, Bo-Ru Lu, Kuan-Po Huang, Wenze Ren, Ming-To Chuang, Chun-Yi Kuan, Jiawei Du, Byeonggeun Kim, Qingming Tang, Chieh-Chi Kao, Ankur Gandhe, Chao Wang, Hung-yi LeearXiv 2026 · arXiv

Evaluating how well an AI generates audio from a text prompt has traditionally been a "black box" affair. You either pay humans to listen and rate clips—which is slow and expensive—or you use automated metrics like CLAPScore that provide a single, often opaque number with mediocre correlation to human judgment.

A new paper from researchers at National Taiwan University and Amazon AGI introduces CoTTAGE (Chain-of-Thought Text-to-Audio Generation Evaluator), a reasoning-based judge that doesn't just give a score; it explains its "thought process" in plain English.

TL;DR

CoTTAGE is an explainable evaluator for text-to-audio generation that follows a "reason-then-predict" paradigm to provide both accurate scores and audio-grounded rationales. By leveraging a novel Thurstone-based reinforcement learning objective and over 10k new human annotations, it achieves state-of-the-art correlation with human judgment while producing explanations that humans actually prefer over those from frontier models.

What Problem Does This Paper Address?

The rapid advancement of Text-to-Audio (TTA) models has outpaced our ability to evaluate them. Current automatic metrics primarily rely on embedding similarities (like CLAP), which fail to capture nuanced temporal misalignments or subtle quality artifacts. Furthermore, these metrics offer no diagnostic feedback. If a model gets a low score, the developer doesn't know why.

While Large Audio Language Models (LALMs) like Gemini or Qwen-Omni can technically "talk" about audio, they often suffer from hallucinations—describing sounds that aren't there or simply repeating the text prompt instead of listening to the audio. This creates a bottleneck where researchers must still rely on labor-intensive human evaluation for reliable results.

Figure 1: An example of text-audio relevance evalua- tion with different models. CLAPScore predicts only a scalar score, while Qwen2.5-Omni often hallucinates reasoning. In contrast, our CoTTAGE model provides an audio-grounded explanation highlighting a temporal- order mismatch.

Methods

CoTTAGE is built on the Qwen2.5-Omni-7B backbone and trained in a sophisticated two-stage pipeline:

  1. Supervised Fine-Tuning (SFT): The model is first trained on "reasoning traces." The authors used Gemini-3.1 Pro to generate rationales for existing datasets, conditioned on the ground-truth human scores. This teaches CoTTAGE the structure of a good evaluation: analyze the audio evidence first, then conclude with a score.
  2. Reinforcement Learning (RL): To further align the model with human preferences, the authors employed Grouped Relative Policy Optimization (GRPO). They introduced a novel Thurstone-based pairwise preference reward. Instead of just regressing to a point estimate, the model generates multiple reasoning trajectories. The reward is calculated by converting the resulting score distribution (mean and variance) into pairwise win probabilities, effectively rewarding the model for being both accurate and certain.
Figure 3: The overall training pipeline of CoTTAGE.

Datasets

The researchers utilized several key benchmarks:

  • RELATE & XACLE: Originally designed for Text Relevance (REL), the authors contributed 10,362 new human annotations for Overall Quality (OVL) to these sets to enable joint training.
  • PAM-Audio & Clotho: Used as out-of-domain test sets to prove the model's ability to generalize beyond its training data.

Key Insights and Ideas

  • Reasoning Improves Accuracy: Forcing the model to generate a rationale before the score (Chain-of-Thought) isn't just for show—it significantly boosts the correlation with human judgments.
  • General LALMs are Poor Judges: Out of the box, even powerful models like Qwen3-Omni-30B show poor correlation with human quality ratings. Audio understanding is a necessary but insufficient condition for audio evaluation; task-specific alignment is critical.
  • Aesthetics $\neq$ Quality: The authors found that existing "aesthetic" metrics (like production quality or enjoyment) are poor proxies for the "Overall Quality" (OVL) humans care about in TTA. This justifies the need for their newly collected 10k+ OVL labels.
  • Uncertainty-Aware RL: By using a Thurstone model in the RL stage, the system naturally yields "softer" rewards when the model is inconsistent across different reasoning paths, leading to more stable and conservative training.

Key Results

CoTTAGE consistently outperformed all baselines, including proprietary frontier models, across six different benchmarks:

  • Superior Correlation: On the XACLE benchmark for text relevance, CoTTAGE achieved a Pearson Correlation (LCC) of 0.685, dwarfing CLAPScore (0.472) and Gemini-3.1 Pro (0.551).
  • Human-Preferred Explanations: In blinded head-to-head tests, human annotators preferred CoTTAGE's reasoning over Gemini-3.1 Pro's in 65.3% of cases for relevance and 53.4% for quality.
  • RL Boost: The reinforcement learning stage provided a consistent gain of roughly 0.05 to 0.08 in correlation metrics over the SFT-only version, proving the value of the pairwise preference objective.
Figure 4: Results of human pairwise evaluation of gen- erated scoring rationales. CoTTAGE is compared against Qwen2.5-Omni-7B, the SFT-only variant, and the pro- prietary model Gemini-3.1 Pro.

Limitations and Future Work

Despite its success, the authors note a few areas for growth:

  • Short-form Focus: The current benchmarks and model are optimized for clips under 30 seconds. Evaluating long-form "soundscapes" remains a challenge.
  • Lack of Precise Timestamps: While CoTTAGE describes what is wrong (e.g., "the snoring starts too early"), it doesn't yet output exact millisecond timestamps for errors.
  • Domain Specificity: The training data is largely environmental sounds. Extending this to music-centric or speech-centric generation is a promising next step.

CoTTAGE represents a significant step toward "Glass Box" AI evaluation, providing TTA developers with a tool that is not only more accurate but also helps them understand exactly where their models are falling silent.

Summary updated

Abstract

Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive human evaluation that bottlenecks model development iteration. To address this, we propose CoTTAGE, a reasoning-based evaluator for text relevance (REL) and audio quality (OVL). Following a reason-then-predict paradigm, CoTTAGE first generates an audio-grounded rationale to diagnose errors before predicting scores. Supported by over 10K newly collected human OVL annotations, CoTTAGE is first trained via supervised fine-tuning and then refined with reinforcement learning using a Thurstone-based preference objective that converts sampled score distributions into pairwise preference probabilities and rewards agreement with rankings derived from human judgments. Across six benchmarks spanning REL and OVL, CoTTAGE achieves state-of-the-art correlation with human judgments while producing human-preferred explanations to accelerate TTA development.
BibTeX
@misc{lin2026cottage,
  author = {Yu-Xiang Lin and Yun-Man Hsu and Bo-Ru Lu and Kuan-Po Huang and Wenze Ren and Ming-To Chuang and Chun-Yi Kuan and Jiawei Du and Byeonggeun Kim and Qingming Tang and Chieh-Chi Kao and Ankur Gandhe and Chao Wang and Hung-yi Lee},
  title = {{CoTTAGE: An Explainable Judge for Text-to-Audio Generation}},
  year = {2026},
  eprint = {2611.17644},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17644},
}
Figure