When a listener judges a three-minute song or a two-minute narration, one crackle, one misplaced stress, or one jarring chord can sink the whole piece, even if the remaining 95% is flawless. Most automatic scores and reward models do the opposite. They average quality over the clip, or summarize it with a single global embedding. This note proposes that part of the well-documented gap between objective audio metrics and human judgment is an aggregation mismatch. Humans score a long piece roughly the way a risk-averse decision-maker scores an investment: by its bad tail, not its mean. Once local quality is aggregated the way listeners do it, two questions follow. Does the aggregated score explain human ratings better than averaging? And can we train generators toward it?
The training question is where a classical problem from risk-sensitive reinforcement learning enters. A tail-weighted aggregate, such as "the average quality of the worst 20% of segments," is not a sum of per-segment rewards. Its optimal decisions therefore depend on history: a generator that has already produced two weak segments should behave differently from one that has not. In the reinforcement learning literature this property is called time-inconsistency, and it breaks the usual step-by-step value recursion. We propose a dense per-segment reward whose sum provably optimizes a well-defined tail-over-time objective. We show that, unlike the decompositions known to be flawed, this one is exact because of how its dual form is structured. We then test whether it beats the obvious alternative of scoring the whole clip once at the end.
Core Claim
The claim has two parts, and the second only matters if the first holds. First, holistic human ratings of long-form audio are better predicted by a lower-tail aggregate of local quality scores (the mean of the worst α-fraction of segments) than by the mean of those scores. Second, for a generator trained with policy-gradient reinforcement learning on long audio, a threshold-based redistribution of this aggregate into per-segment rewards gives faster and more reliable reduction of local defects than either a terminal-only tail score or a dense mean score. The advantage should grow with the number of segments.
The mechanism rests on one identity. Let $r_1,\dots,r_K$ be quality scores of the $K$ segments of a generated clip $\tau$, and let $\alpha\in(0,1]$ be the tail fraction. The tail-over-time score is the Rockafellar–Uryasev form of conditional value-at-risk (the expected value of the worst α-fraction):
$$ C_\alpha(\tau) \;=\; \max_{b}\; \Big\{\, b \;-\; \frac{1}{\alpha K}\sum_{k=1}^{K} (b - r_k)^+ \Big\}, $$
where $b$ is a threshold and $(x)^+=\max(x,0)$. For a fixed $b$, the bracket is a sum of per-segment terms $b/K - (b-r_k)^+/(\alpha K)$. These terms penalize only segments that fall below the threshold, so they form a legitimate additive dense reward.
The key structural point concerns the maximization over $b$. On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processes shows that standard risk-level decompositions of CVaR fail because they swap the policy's maximization with a minimization over the risk dual. That same paper also shows value-at-risk decomposes exactly because its dual is a supremum that commutes with the policy maximum. Our objective has the same favorable sign: $\max_\pi \max_b$ commute. Jointly optimizing the policy and a per-prompt threshold therefore exactly solves the "pooled" objective $\max_b \mathbb{E}_\tau[\cdot]$, which is the CVaR of segment quality pooled across rollouts.
What we actually care about is the per-clip objective $\mathbb{E}_\tau[C_\alpha(\tau)]$. This differs from the pooled objective by the gap $\mathbb{E}\max_b - \max_b\mathbb{E}\ge 0$. That gap is the time-inconsistency in this setting, and it is measurable: it is large exactly when different rollouts for the same prompt have very different quality quantiles. Measuring it, and correcting for it when it is large, is part of the program.
Relation to Prior Work
The motivation comes from the audio side. Three papers document that clip-level summaries miss what listeners care about. Diagnose, Then Refine shows that utterance-level metrics miss local prosodic defects. It repairs them post hoc with an AudioLLM judge and a separate refiner model, whereas we train the generator to avoid such defects in the first place. SongEval supplies full-length songs with professional holistic ratings and finds generic aesthetic scorers correlate only modestly with them. A Benchmarking Initiative for Audio-domain Music Generation argues that distribution metrics like Fréchet Audio Distance are blind to temporal structure. None of the three asks how local quality should be combined into a whole-piece score.
On the generator-training side, three lines of work are risk-neutral over time. MusicRL fine-tunes MusicLM with a KL-penalized expected clip-level reward. GROW normalizes clip-level rewards within groups of rollouts to fight reward saturation in text-to-speech. DRAGON optimizes distributional rewards over batches of outputs. DRAGON is closest in spirit, since it also moves beyond per-sample expectation. However, its "distribution" is across different generated pieces, while ours is across time within one piece, which is what creates the sequential credit-assignment problem.
Generative Adversarial Post-Training constrains the design. It reports that optimizing a single musical objective collapsed a jamming policy into trivial chords, and that KL penalties did not stop this. A tail objective could worsen that collapse by rewarding safe blandness, so diversity is a first-class metric here.
The method comes from risk-sensitive RL. Reward Redistribution for CVaR MDPs turns a terminal CVaR into dense per-step rewards on an augmented problem with a contracting Bellman operator, but only on tabular gridworlds. We reuse the redistribution idea with a different target. Their target is tail risk over random outcomes. Ours is a tail over time inside a single realized trajectory, applied to high-dimensional generators. Beyond Average Reward in Markov Decision Processes asks which trajectory functionals dynamic programming can handle at all, and our threshold form is a concrete instance worth placing in that taxonomy. Recursive time-consistent risk measures, as in Regret Bounds for Risk-sensitive RL with Lipschitz Dynamic Risk Measures and Stochastic Optimal Control with Dynamic, Time-Consistent Risk Constraints, sidestep inconsistency by changing the objective to a nested one. We deliberately do not do this. Listeners judge the static whole-piece tail, so the nested measure would be a different target.
Research Program
The first stage asks whether listeners actually aggregate local quality this way. On SongEval we will slice each song into fixed windows of about 10 seconds. Each window is scored with an off-the-shelf local quality model that was not trained on SongEval's holistic labels; Audiobox-Aesthetic is a natural choice. We then regress the holistic human scores on several aggregates of the window scores: the mean, the minimum, the lower-tail $C_\alpha$ for a sweep of α, a fitted spectral weighting (a learned weight per rank position), and an end-weighted aggregate as a control for recency effects. The same analysis will be run on long-form speech, using LoopTTS-style material where local defects are flagged, with holistic MOS collected on multi-sentence passages. If the mean wins, the rest of the program is not worth running, and that is a useful negative result in itself.
The second stage builds the training signal. For each prompt we draw a group of $G$ rollouts, as in GROW, and score every segment with the local scorer. We set the per-prompt threshold $b$ to the pooled α-quantile of segment scores across the group. Each segment then receives the redistributed reward $-(b-r_k)^+/(\alpha K)$. These rewards enter a group-relative policy-gradient update with per-segment reward-to-go and a standard anchor to the pretrained model.
Because $b$ is computed per prompt-group rather than per clip, the per-segment rewards are additive and causal. Each segment's credit then depends only on that segment and the ones after it. The resulting update is an unbiased gradient of the pooled objective, by the sup-commutation argument above. We will also implement two other variants. The "hindsight" variant sets $b$ to each clip's own quantile. It matches $C_\alpha(\tau)$ exactly, but it biases the reward-to-go estimator because past rewards now depend on future actions. The "corrected" variant adds the terminal residual $C_\alpha(\tau) - \sum_k(\text{redistributed terms})$ as a final reward, which removes the gap between the pooled and per-clip objectives at the cost of some variance.
The third stage is analysis. We will measure the time-inconsistency gap $\mathbb{E}\max_b - \max_b\mathbb{E}$ during training and relate it to how much the corrected variant helps. We expect this gap to grow as rollouts within a group become more heterogeneous. We will also track whether tail optimization trades diversity for safety, the collapse GAPT warns about.
Baselines
The naive reference is the pretrained generator with no reinforcement fine-tuning, sampled with default settings. Every other method is reported as a change relative to it.
The strongest existing training method is group-relative policy gradient on clip-level rewards, implemented following GROW. We run it in two forms. The first uses its usual clip-level quality and intelligibility rewards. The second uses the terminal tail score $C_\alpha(\tau)$ as a single end-of-clip reward. The terminal form is the most dangerous competitor, because it optimizes exactly the per-clip objective and differs from our method only in credit assignment. For reference, GROW reports word error rate falling from 2.016 to 1.558 and speaker similarity rising from 0.676 to 0.715 on its text-to-speech setup. Any variant of ours must at least preserve those levels while reducing local defects.
For music, the relevant reference is MusicRL, which reports MOS rising from 3.07 to 3.82 and an 87% preference rate (including ties) over MusicLM. MusicLM itself may not be available. We will therefore reimplement its recipe, a KL-penalized expected clip-level reward, on whichever open text-to-music model we use, and compare against that rather than its reported numbers. As a further distributional reference we will consider DRAGON, which reports an 81.45% average win rate across 20 rewards.
The ablation that isolates our key ingredient is a dense mean reward: each segment receives its own score $r_k$. This has the same density and credit structure as our method but is risk-neutral over time, so it separates "dense credit" from "tail target." The hindsight and corrected variants form a second ablation that isolates the time-inconsistency gap.
All methods share the same base model, prompts, group size, number of rollouts, local scorer, and anchor strength. Each gets an identical hyperparameter search budget. Beating the strongest baseline means two things. At matched rollout budget, our method must achieve a lower human-flagged local-defect rate and a higher worst-segment MOS than terminal-tail GRPO, without losing holistic MOS, word error rate, or Vendi diversity. And this margin should widen as clip length increases.
Why not just…?
Why not just use the tail score as a terminal reward with standard GRPO? This alternative optimizes the right objective and is a genuine threat. Its weakness is credit assignment. A single scalar at the end of a clip with $K$ segments tells the generator neither which segment was bad nor whether it was bad at all. The variance of that signal should grow with $K$, because every segment's actions share one noisy score. Our horizon-scaling experiment settles this directly. If terminal GRPO matches the dense redistributed reward even at the longest clips, we will report that the contribution reduces to the aggregation finding (a better reward, not a better optimizer) and drop the redistribution claim.
Why not just average per-segment scores densely, with a KL penalty? That reward optimizes the mean, which the first stage is designed to show is the wrong target. It also lets the generator buy average quality while leaving rare bad moments untouched. GAPT's finding that KL penalties did not prevent collapse in Generative Adversarial Post-Training suggests the KL term will not supply the missing tail sensitivity. The dense-mean ablation tests this claim directly.
Why not just generate N candidates and keep the one with the best tail score, or repair defects after the fact? Best-of-N with a tail-aware scorer is cheap to try, and we include it at matched compute. Two pieces of evidence count against it at long horizons. Regenerating whole clips is hit-or-miss for local defects: Diagnose, Then Refine reports targeted repair beating regenerate-from-scratch 77% of the time. And the cost of full-length candidates scales with clip duration. Post-hoc repair in the LoopTTS style is strong (85.7% preference over defective audio), but it requires an AudioLLM judge in the loop at inference time. We compare against LoopTTS as an upper reference for repair quality, not as a cost-matched baseline. The interesting outcome would be a fine-tuned generator that needs repair less often.
Proposed Experiments
The aggregation test on SongEval is the first decisive check. SongEval reports Pearson correlations of 0.61–0.66 for Audiobox-Aesthetic and 0.912 for a UTMOS-style model trained on SongEval itself. Our question is whether windowed Audiobox-Aesthetic scores, aggregated by $C_\alpha$ with α fitted on held-out songs, move clearly above the 0.61–0.66 range while mean aggregation stays near it. Matching 0.912 is not the bar, since that model is trained on the target labels. We will repeat the test on the speech material, and we require the preferred α to be stable across genres and languages before trusting it as a training target.
Training experiments start with long-form text-to-speech, where rollouts are cheap and defect labels exist. We will use LibriTTS and Seed-TTS-style prompts concatenated into passages of 2, 4, 8, and 16 sentences, with an open flow-matching or autoregressive-diffusion text-to-speech model of the kind GROW fine-tunes. The second domain is text-to-music clips of increasing length, evaluated with the SongEval scorer and CMI-RewardBench's reward model as independent judges. That reward model reaches 78.20% musicality accuracy on its benchmark, against 65.80% for Gemini 3 Pro. Neither judge is used for training, which guards against the generator simply hacking the training scorer.
Our metrics fall into four groups. The first is human-flagged local-defect rate, worst-segment MOS, and holistic MOS, from blind listening tests. The second is word error rate and speaker similarity for speech. The third is Vendi diversity, used to detect collapse toward safe outputs. The fourth is learning curves of defect rate against number of rollouts.
The second decisive check is horizon scaling: the gap between dense redistributed rewards and terminal-tail rewards should be near zero at 2 sentences and grow with length. The third is the time-inconsistency diagnostic. The corrected variant should help in proportion to the measured gap $\mathbb{E}\max_b-\max_b\mathbb{E}$. If the gap stays negligible, the pooled-threshold reward alone suffices, and the history-dependence that motivates the work is weak in practice.
References
- On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processes
- Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset
- MusicRL: Aligning Music Generation to Human Preferences
- GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech model
- DRAGON: Distributional Rewards Optimize Diffusion Generative Models
- Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction
- Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity
- Beyond Average Return in Markov Decision Processes
- Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measures
- Stochastic optimal control with dynamic, time-consistent risk constraints