Automatic scores for generated music and speech often disagree with human listeners. A model can score well on signal-level or statistical metrics and still sound incoherent, or score poorly and sound excellent. The usual responses sit at two extremes. One is a large toolkit of dozens of metrics that the user must choose among and configure by hand. The other is a multi-billion-parameter audio language model that writes a critique before giving a score. We propose a middle path. We learn, from human preference data, a short and readable recipe: a handful of existing fast metrics, with weights that depend on the text prompt, that together predict which of two audio clips a human prefers. The user states only the prompt. The system returns the few metrics that matter for that prompt and how to combine them.
The two research directions meet in one question: how many knobs does perceptual evaluation actually need? Software toolkits that expose every component, such as RETURNN, MMOCR and TorchRL, document that fine-grained configurability raises the barrier to entry and pushes practitioners back to simple scripts. In audio evaluation, that simple script is exactly the flawed single metric (SNR, FAD) that diverges from perception. If a learned, prompt-conditioned selection of three to eight metrics recovers most of the human agreement of heavyweight judges, then the divergence is largely a problem of selection and context, not missing signal. The recipe also becomes the low-code interface that the toolkits lack.
Core Claim
We claim that existing fast audio metrics already carry most of the signal needed to predict human preference, but they are applied uniformly regardless of context. A prompt like "lo-fi" should turn a hiss penalty into a neutral or positive feature. Chromatic deviation in jazz should not be treated as an outlier, as Improving Automatic Jazz Melody Generation by Transfer Learning Techniques found statistical metrics do.
Concretely, for audio $x$ and prompt $c$, we score
$$ s(x, c) = \sum_{k=1}^{M} g_k(c)\, \phi_k\big(m_k(x, c)\big), \qquad \|g(c)\|_0 \le K, $$
where $m_k$ is the $k$-th of $M$ metrics from a fixed bank, $\phi_k$ is a learned monotone calibration of that metric, $g(c)$ is a gate computed from a text embedding of the prompt, and $K$ is a sparsity budget on how many metrics may be active. We train with a Bradley–Terry pairwise loss, $-\log \sigma(s(x_w,c) - s(x_l,c))$, where $x_w$ and $x_l$ are the preferred and dispreferred clips. Each aesthetic dimension (musicality, alignment, naturalness) gets its own gate.
Sweeping $K$ traces an explicit curve of recipe length against agreement with humans. That curve is the quantitative version of the modularity–usability trade-off that the software literature describes only qualitatively.
Research Program
The metric bank is built on VERSA. VERSA bundles 65 metrics with more than 700 variants in four families (reference-free, sample-aligned, unaligned-reference, and distributional), is configured through YAML, and itself reports that SNR diverges from perceived quality. VERSA supplies the method's raw material. We differ from it by learning which metrics to run and how to weigh them, rather than leaving that to the user.
Generative outputs have no sample-aligned reference, so we exclude the sample-aligned family. That family is the one ViSQOL v3 shows breaks on generative codecs, because tiny time shifts register as large errors. We keep the reference-free metrics (for example NISQA-style quality predictors), the prompt-conditioned ones (CLAP score), and per-sample versions of distributional metrics. The per-sample version is the embedding distance from the clip to a genre-matched reference set. We know from Fréchet Audio Distance and the FreeSound Loop benchmark that these distributional features capture surface texture rather than meter or harmony. That finding is a constraint: it predicts they will earn nonzero weight for timbre-heavy prompts but not for structure-heavy ones, and we will check this. To cover rhythm and structure, which the bank lacks, we add a small number of cheap music-information features such as beat-tracking stability and self-similarity-based repetition. The gate is free to ignore them.
Training data comes from the benchmarks in the file. For music we use the CMI-Pref training split, with its expert pairs, plus the 110,000 pseudo-labeled CMI-Pref-Pseudo pairs from CMI-RewardBench, and SongEval's 2,399 expert-rated songs from SongEval, converted to pairs. For speech we use SpeechJudge-Data from SpeechJudge.
Beyond accuracy, the research program analyzes three properties that a single neural score cannot expose. The first is context flips: we will test whether the learned gate reverses the sign of noise-related metrics for lo-fi and distortion prompts, which is the effect CMI-RewardBench reports when prompts contextualize quality. The second is disentanglement. CMI-RM's musicality and alignment heads correlate at SRCC ~0.85 despite separate architecture. We measure whether per-dimension recipes select overlapping metrics and whether their scores correlate less, which bears directly on SongEval's reported entanglement across dimensions. The third is auditability. Because the recipe is readable, we can design attacks from it: audio that maximizes the active metrics while being bad. We then measure how quickly the recipe can be patched compared with a black-box reward.
Baselines
The naive reference point is the single most popular fixed metric per task, with no learning. For music that is CLAP score or per-sample FAD-embedding distance; for speech it is a reference-free MOS predictor. We also include an equal-weight average of all bank metrics after per-metric standardization. This shows how much of any gain comes from selection and context rather than from simply having many metrics.
The strongest fast learned competitor for music is CMI-RM from CMI-RewardBench. It is a prompt tower, audio tower and joint transformer trained on the same CMI-Pref data. It reports 78.20% accuracy on the CMI-Pref musicality task and 73.43% on Music Arena, against 65.80% for Gemini 3 Pro. It is the right comparison because it is single-pass, prompt-aware and trained on identical labels. We will use its released numbers and, where weights allow, rerun it on our splits. Beating it means matching its in-domain accuracy within confidence intervals while exceeding 73.43% on Music Arena held out from training. We expect fixed, hand-designed metrics to overfit less across benchmarks.
For song aesthetics, SongEval reports that a UTMOS-based model trained on SongEval reaches Pearson 0.912 with human averages, while Audiobox-Aesthetic sits at 0.61–0.66. On Music Arena's 733 song pairs, MuseCritic reports 71.35% pairwise accuracy, the highest among its compared models. MuseCritic is an 8B critique-then-score judge. For speech, SpeechJudge-GRM reports 77.2% on SpeechJudge-Eval (79.4% with 10-way voting), against 72.7% for a Bradley–Terry scalar model and 69.1% for Gemini-2.5-Flash.
The Bradley–Terry scalar model is the fair same-data comparator. We will train our own Bradley–Terry head on a frozen audio encoder using the same pairs, the same hyperparameter-search budget and the same seeds as our recipe learner. The large critique judges serve as ceilings. Against them, the claim is agreement within a few points at orders of magnitude lower wall-clock cost, which we will measure on shared hardware rather than assert.
Two ablations isolate the key ingredient. An unconditioned sparse recipe, with the gate fixed across prompts, tests whether context matters. A recipe learner given the frozen-encoder embedding as one extra "metric" tests whether the fixed metrics add anything beyond a learned embedding. If the gate routes all weight to that embedding, our claim fails.
Why not just…?
Why not just train a scalar regressor on audio embeddings, like the UTMOS model that reaches 0.912 on SongEval? This is the most serious threat. On in-domain correlation it may simply win. MuseCritic reports that such regressors compress predictions toward the mean, and SpeechJudge shows a Bradley–Terry scalar model trailing the critique judge by 4.5 points, but neither result shows scalar models losing to sparse metric combinations. Our case rests on cross-benchmark transfer (train on CMI-Pref, test on Music Arena and SongEval), on the context-flip and disentanglement analyses, and on resistance to reward hacking under large best-of-N. If the embedding regressor matches us on transfer and on hacking resistance, the only surviving contribution is readability, and we will report it that way.
Why not just ask a language model to write the VERSA configuration from the user's intent? That would deliver the low-code interface without training. But nothing grounds its choices in human preference, and frontier multimodal models are weak judges here: below 80% on CMI-Pref in CMI-RewardBench, below 70% on speech naturalness in SpeechJudge. We include this as a baseline. An LLM selects $K$ metrics per prompt and weights them equally, and we compare it with the learned gate at the same $K$. A small gap would mean preference data adds little and that the paper's real contribution is the metric bank.
Why not just distill MuseCritic or SpeechJudge into a small fast head? Distillation inherits the teacher's biases, including its bilingual, vocal-track focus. It also produces another opaque score, so the auditing and patching analysis becomes impossible. We will still run it, training a distilled head on critique-judge scores over the same audio, because it is the natural route to closing the latency gap. If it beats our recipe on transfer, it should be preferred for pure scoring.
Why not just use FAD or CLAP, which are already automatic? The FreeSound Loop study shows FAD rewards surface texture over meter, and CMI-RewardBench shows FAD cannot score individual samples while CLAP fails on compositional instructions. These metrics remain inside our bank as candidates. The gate decides when they deserve weight.
Decisive Experiments
The first check is the complexity–agreement curve. We sweep $K$ from 1 to $M$ on the CMI-Pref test split, SongEval and SpeechJudge-Eval, and report the smallest $K$ that reaches within two points of the full-bank learner and of CMI-RM. If no $K \le 8$ gets within five points of CMI-RM, the "short recipe" thesis is false.
The second check is transfer. Recipes trained only on CMI-Pref data are evaluated on Music Arena pairs, compared against CMI-RM's 73.43% and MuseCritic's 71.35%, and tested under the unconditioned-gate ablation.
The third check is context. On style prompts containing lo-fi, distortion or jazz cues, we test whether the gate's weight on noise and outlier metrics changes sign. On matched pairs where only the prompt changes, the conditioned recipe should flip its preference and the unconditioned one should not.
The fourth check is disentanglement: we report SRCC between the musicality and alignment recipe scores against CMI-RM's ~0.85.
The fifth check is downstream reranking and hacking. We rerank candidates from an open music generator at best-of-10, the protocol CMI-RewardBench confirmed with human A/B tests, and push to best-of-100. Blinded listeners then judge whether each reward's top pick keeps improving or starts to degrade. We also run red-team clips constructed from the published recipe. In the same study we measure per-clip evaluation latency for every method on identical hardware.
References
- RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
- MMOCR: A Comprehensive Toolbox for Text Detection, Recognition and Understanding
- TorchRL: A data-driven decision-making library for PyTorch
- Improving Automatic Jazz Melody Generation by Transfer Learning Techniques
- VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music
- ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
- Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms
- A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
- MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques