Forecasting Perceptual Failure From the First Seconds of a Generated Song

Generative music and speech systems are checked for quality the way software used to be: at the end. A model writes a full three-minute song, a learned judge scores it, and if the song is bad it is thrown away. Best-of-N sampling generates N complete songs to keep one. Software engineering moved away from this pattern long ago. Under "shift-left testing," risk is predicted at each small code change, where fixing it is cheapest, rather than at the release gate. We propose the same move for audio generation. The idea is a small verifier that reads a partial generation, meaning the prompt plus the first few bars or seconds, and predicts how a human-aligned judge would score the finished piece. Doomed rollouts can then be stopped early, and the compute goes to promising ones.

The difficulty, and the reason this is a research question rather than an engineering one, is that perceptual and objective quality diverge. The cheap signals that are easy to compute on a prefix, such as spectral distance, transient detectors, or utterance-level quality predictors, are the ones known to disagree with listeners. They penalize lo-fi texture, intentional glitches, and adventurous harmony. Structural failures like losing the meter or never returning to the chorus also do not exist in a prefix yet. So an early check must forecast a future perceptual outcome, not measure present signal quality. Whether that forecast is possible, and how early, is the central unknown.

Core Claim

A verifier trained to regress the terminal judge's score of the full rollout from its prefix can prune a population of partial generations. At equal generation compute, this should select outputs that listeners prefer over those chosen by Best-of-N with the terminal judge. It should also beat two alternatives: applying that same judge to truncated audio, and cheap objective gates. Concretely, for prompt $c$ and partial generation $x_{1:t}$, the verifier estimates

$$ v_\phi(x_{1:t}, c) \approx \mathbb{E}\left[\, R(x_{1:T}, c) \mid x_{1:t}, c \,\right], $$

where $R$ is a terminal perceptual judge applied to the completed generation $x_{1:T}$, and the expectation is over the generator's own continuations. This quantity differs from "how good does the prefix sound." Most of the proposal is built to test whether that difference matters.

We also borrow a second idea from commit-level defect prediction. Many generation failures are mismatches between intent and execution: the prompt asked for a tempo, key, or instrumentation that the prefix already violates. We add an auxiliary objective that trains the verifier to detect a swapped prompt, and an ablation measures its contribution.

Relation to Prior Work

The motivating evidence comes from four sources. Diagnose, Then Refine builds a filter–judge–refiner loop for speech that uses Gemini-3-pro as critic. It shows that utterance-level metrics (word error rate and UTMOS) miss local prosodic defects. The whole loop runs after full synthesis, and its frontier-model critic is far too slow to sit inside generation.

CMI-RewardBench trains a compact two-tower music reward model, CMI-RM, and uses it for Best-of-N selection confirmed by human A/B tests. It discards N−1 finished songs, and its musicality and prompt-alignment heads are entangled, with a rank correlation of about 0.85. SpeechJudge trains a reasoning-based naturalness judge and names whole-utterance granularity as its main limitation. GROW pushes perceptual rewards into the text-to-speech model's weights through reinforcement learning, which changes the generator rather than monitoring it at inference.

None of these four evaluates or intervenes on partial generations, and that is the gap we target. We use CMI-RM, SongEval, and SpeechJudge as the terminal judges $R$ that our verifier distills. They supply the target, not the competition.

Two papers constrain the design. Audio Defect Detection in Music with Deep Networks shows that a rule-based click detector falls to 80.34% precision on electronic music because intentional transients look like defects. ViSQOL v3 documents how sample-aligned metrics punish phase and timing differences that listeners cannot hear. Together they rule out an objective-signal gate as the early oracle, and we test that rather than assume it. Improving Automatic Jazz Melody Generation adds a sharper warning: statistical evaluators treated chromatic enclosures as outliers, yet experts preferred them. Any early gate risks the same conservatism.

Three papers supply methods. The Replaced Message Identification objective of JIT-BiCC trains a model to tell whether a commit message matches its code change; we reuse it as prompt–prefix mismatch detection. Agogic contributes performance-timed music tokens and a clean autoregressive testbed, where the prefix is well defined. It also finds that text-to-music models "imprint" their training distribution and ignore prompt constraints, a failure that should already be visible early. SongEval supplies an aesthetic predictor with Pearson 0.912 against expert ratings.

Research Program

The primary testbed is text-to-symbolic music. We train a 0.8B autoregressive model with performance-timed tokens on MidiCaps following the Agogic recipe, and render its outputs to audio for judging. This setting makes prefixes exact and rollouts cheap. For each of several thousand prompts, we sample many full generations. At checkpoints of 5%, 10%, 25%, and 50% of the target length, we branch several independent continuations from each prefix. We then score every completed piece with CMI-RM (musicality and alignment heads) and with a SongEval-trained predictor. This yields Monte Carlo estimates of the expected terminal score for each prefix, which serve as regression targets for $v_\phi$.

The verifier is a small transformer over the prompt and prefix tokens, meant to run in milliseconds. It is trained on these targets plus the swapped-prompt auxiliary task. A second testbed checks transfer: an autoregressive text-to-speech model judged by SpeechJudge-GRM, where the critical failures are local prosodic ones.

Inference uses successive halving. We start $K$ prefixes, keep the top fraction by $v_\phi$ at each checkpoint, and complete only the survivors, with $K$ chosen so total generated tokens match Best-of-N. The analysis side asks three questions. How does the verifier's predictive power (rank correlation with realized terminal score) grow with prefix length? Which failure types, whether prompt violation, loss of meter, or harmonic drift, become predictable earliest? And does the forecasting verifier differ from a present-quality scorer in what it rejects?

Baselines

The naive reference is a single sample with no selection. It anchors both the quality scale and the compute scale.

The strongest existing method is Best-of-N with the terminal judge: generate N full pieces and keep the one CMI-RM ranks highest, as in CMI-RewardBench. It is the strongest competitor because it uses the most informative judge on complete outputs. The file reports CMI-RM at 78.20% agreement with experts on the musicality task, against 65.80% for Gemini 3 Pro. No reported result found on Lacuna for the size of CMI-RM's Best-of-N gain in human win rate, so we measure it ourselves. It runs on the same generator, prompts, and token budget. We sweep N and report the full quality–compute curve. Beating it means one of two outcomes. Either our method reaches Best-of-N's human preference at half the tokens or less, or it wins a blind human A/B test at matched tokens, with the confidence interval excluding 50%.

The second baseline is the same terminal judge applied to rendered prefix audio in the same successive-halving schedule, with no new training. It isolates the key ingredient, forecasting the future outcome versus scoring the present. We also run a cheap objective gate: Fréchet-distance-style statistics plus a transient-defect detector on prefixes. This measures how badly perceptual–objective divergence hurts early pruning.

The ablations remove the swapped-prompt objective. They also replace the Monte Carlo continuation targets with the prefix's own judge score, and vary checkpoint placement. For speech, the corresponding reference is SpeechJudge-GRM at 77.2% accuracy on SpeechJudge-Eval, used for Best-of-N.

Why not just…?

Why not just run CMI-RM or SongEval on the truncated audio? This is the real threat. If prefix quality predicts final quality well, our trained verifier adds only speed. We think it will fall short for structural properties: SongEval's coherence and structural-clarity dimensions are defined over whole songs, so a strong opening can precede a formless middle. It will also fall short for failures that are visible early in tokens but not yet audible, such as tempo drift relative to the prompt. The truncated-judge baseline in the same pruning schedule settles this directly. If it matches our verifier at every checkpoint, we will report that. The contribution then becomes the compute-scheduling result and the prefix-predictability curves, not the verifier.

Why not just use smaller N with ordinary Best-of-N? A cheaper Best-of-N is a point on the same curve we sweep. Early pruning helps only if bad rollouts are identifiable before much compute is spent. The quality–compute frontier comparison shows directly whether our curve lies above Best-of-N's. The predictability-versus-prefix-length analysis explains why or why not.

Why not just gate on cheap objective metrics at each step? The evidence argues against it. Rule-based click detection falls to 80.34% precision on electronic music (Hook-Net paper). Audiobox-Aesthetic reaches only Pearson 0.61–0.66 with human aesthetic ratings against 0.912 for a trained predictor (SongEval). Still, we run this gate and report its per-style rejection rates, because the harm we expect is specific: disproportionate pruning of lo-fi, glitch, and jazz prompts.

Why not just train the generator with GROW-style reinforcement learning so bad rollouts stop happening? Weight-level alignment and inference-time pruning are complementary, and GROW's reported gains are in word error rate and speaker similarity rather than structural musical failures. We apply early pruning on top of a reward-fine-tuned generator. If the gain vanishes once the generator is aligned, that is an important negative result.

Decisive Experiments

The first check is predictability. On held-out prompts, we plot the rank correlation between $v_\phi$ and the realized terminal judge score against prefix fraction, alongside the truncated-judge baseline. The idea is alive only if the correlation at 10–25% of length is high enough that pruning at that point loses few of the rollouts Best-of-N would have chosen. We measure this as the recall of Best-of-N's winner among survivors.

The second check is the matched-compute human study. Expert listeners compare our selection against Best-of-N on about 300 prompts spanning genres, in blind A/B trials. This matters because both judges agree with humans only about 78% of the time, so gains measured by the judge alone could be judge-hacking. We report human win rate and the gap between judge-measured and human-measured gains.

The third check probes divergence. We stratify pruning decisions by prompt style (lo-fi, electronic, jazz, conventional pop) and have listeners rate a sample of rollouts that each method rejected. An early gate that disproportionately discards pieces humans like has reproduced perceptual–objective divergence one level upstream. We expect this failure for the objective gate. It must not appear for the forecasting verifier.

The speech replication with SpeechJudge-GRM tests whether the result carries over to a domain where failures are local rather than structural. Our prediction is that it largely does not, since a local glitch can occur anywhere and is not forecastable from a prefix. That prediction bounds where shift-left verification of generation is worth doing.