When a music generator is paired with an automatic judge, the simplest way to get better songs is to generate several candidates and keep the one the judge likes best. Drawing more candidates means pushing harder against the judge. Work on text models shows that past some point, the judge's favorite stops being what people prefer: true quality rises, peaks, then falls. Automatic audio judges are especially easy to fool, because the scores they produce often diverge from what listeners hear. We propose to treat the number of candidates as a drug dose and to set it separately for each prompt, using the rule-based escalation logic of first-in-human cancer trials. The budget doubles only while a separate "safety monitor" judge, never used for selection, confirms that the selected clips are not reward hacks. The monitor is a judge from a different model family, so its errors should differ from the selector's. Escalation stops once two of the top candidates trip that alarm.
The opportunity matters because the field is currently scaling inference-time search on music with no stopping rule. CMI-RewardBench reranks a fixed 10 candidates with a single reward model and reports human-preferred outputs. Scaling Laws for Reward Model Overoptimization predicts that true reward under best-of-$n$ follows $R(d)=d(\alpha-\beta d)$, where $d$ grows with $n$, $\alpha$ is how well the judge tracks the target, and $\beta$ is how fast it is exploited. If $\alpha$ and $\beta$ differ from prompt to prompt, as they plausibly do across lo-fi, jazz and pop requests, then any single fixed $N$ wastes compute on some prompts and overdoses others.
Core Claim
Our claim is that per-prompt budget escalation can match or beat the best fixed budget at lower average compute. The escalation runs on a conservative judge ensemble for selection and a held-out, heterogeneous critic as a cheap, label-free toxicity signal. It should also cut the rate of catastrophic outputs, meaning songs listeners rate in the bottom quartile, below that of the best fixed budget.
The concrete mechanism borrows the classical 3+3 rule used in dose-finding trials such as the LXI-15029 phase I study. In that rule, three patients are treated at a dose level. Zero dose-limiting toxicities means the dose goes up. One toxicity means three more patients are treated at the same level. Two or more means the trial stops and the previous level is kept.
Our translation works as follows. For a prompt, candidate pools grow by doubling, $N\in\{2,4,8,\dots,128\}$, and each new level reuses the clips already drawn. The selector is a conservative ensemble that scores each clip by the minimum across its members. At each level, the selector's top three clips form the "cohort." A clip counts as a "dose-limiting toxicity" if the held-out monitor ranks it in the bottom half of the current pool. That pattern, where the selector loves a clip and an independent judge rates it below the pool median, is the signature of a reward hack.
The rest of the 3+3 logic carries over directly. With 0 of 3 toxicities, the budget escalates. With 1 of 3, the selector's ranks 4 through 6 are checked as well. With 2 or more, the procedure outputs the selection from the previous level. Budgets may only move to the adjacent level, never skip ahead. This mirrors the nearest-neighbour restriction that Xiao and Zhang's combination-dose CRM uses to avoid toxic leaps on a dose grid.
Because the cohort is drawn from clips already generated, the monitor adds only scoring cost, not generation cost.
Research Program
The work has three parts.
The first part asks whether the hump exists in music at all. Kostric et al. (Reward Model Ensembles Help Mitigate Overoptimization) found the hump in text and showed that worst-case ensembles nearly remove it under best-of-$n$. That result is the strongest prior against our premise, and it has to be tested in audio first. We build a Gao-style controlled setting with a gold judge and weaker proxy judges. The gold judge is trained on all of SongEval (SongEval) plus CMI-Pref. Proxy judges are trained on subsets with 25% injected label noise, following Kostric et al. For each prompt we then trace gold reward against $N$ for a single proxy, a mean ensemble and a worst-case ensemble.
The second part characterizes how much the best budget varies across prompts. We measure the distribution of each prompt's peak budget $N^*$ under the gold judge, and how much of that variation genre or prompt features explain. If $N^*$ is nearly constant, a single tuned budget suffices and our proposal reduces to tuning. We would report that outcome and stop there.
The third part builds and evaluates the escalation rule, including a population-level extension. Within each genre, consecutive prompts start at a budget set by a Dixon up-and-down rule: one level higher if the previous prompt escalated to the cap, one lower if it stopped at its first level. This follows how the oliceridine up-and-down trial located a median effective dose in 31 patients without overshooting. The extension aims to save the low-level probing cost on easy genres.
The heterogeneity of the judges is deliberate. The selector ensemble combines SongEval-trained quality predictors built on different self-supervised audio encoders, an approach that SongEval showed reaches 0.912 Pearson correlation with human ratings in its UTMOS-based variant. The monitor comes from a different family trained on different data. The primary monitor is the prompt-conditioned CMI-RM, chosen because CMI-RewardBench found that context-blind judges unfairly penalize intentional lo-fi styles. As a secondary monitor we use a critique-then-score AudioLLM judge following the MuseCritic recipe. The reasoning comes from the ensembling literature, for example the RoBERTa+ELECTRA ensemble and stacked yield prediction: models only protect each other when their errors differ. A seed-only monitor would share the selector's blind spots.
For generation we use an open song generator from SongEval's source set, such as DiffRhythm or YuE, at roughly 60-second clips to keep 128-candidate pools affordable. That is about 300 prompts across SongEval's nine genres in English and Mandarin.
Baselines
The naive reference point is a single sample with no judge. It establishes whether search helps at all, and it costs one generation per prompt.
The closest published system is single-judge fixed best-of-10 reranking with CMI-RM, as in CMI-RewardBench. That paper reports human A/B gains, but on Lacuna no reported win-rate is found for its best-of-10 reranking. Its judge's reported quality is 73.43% pairwise accuracy on Music Arena and 78.20% on the CMI-Pref musicality task. MuseCritic reports 71.35% on Music Arena. We reproduce best-of-$N$ with CMI-RM at $N=10$ and across the full $N$ grid on the same prompts and generator.
The strongest competitor is a fixed-budget worst-case ensemble in the style of Kostric et al. It uses the same selector ensemble as our method, with $N$ tuned on held-out prompts to maximize gold reward. This is the strongest baseline because in text it virtually eliminated best-of-$n$ overoptimization with no extra hyperparameters. It is the one method that could make our proposal unnecessary. For a fair comparison, it receives the same generator, the same judges, and an identical tuning-prompt budget. We compare at matched mean samples per prompt and also at its own optimal budget.
Beating it means three things together. Our method must match or exceed its human pairwise win rate at no more than its compute, or match it at substantially less compute. It must also have a lower catastrophic-output rate. As a ceiling, we add an oracle that picks each prompt's gold-optimal $N$.
Three ablations isolate the key ingredient. The first replaces the heterogeneous monitor with a seed-only copy of a selector member. The second removes the monitor and stops on the selector ensemble's own disagreement. The third removes the adjacent-level restriction.
Why not just…?
Why not just use a fixed-$N$ worst-case ensemble, since it already beat overoptimization in text? This is the real threat. If Part 1 shows a flat gold curve for the worst-case ensemble in music, escalation cannot improve quality. It can then only save compute. We would still measure that saving, but our quality claim would fall. If the hump persists in audio, a fixed $N$ cannot track prompt-specific peaks. One reason it might persist is that audio judges can be gamed through low-level acoustic texture, a failure that the FreeSound Loop benchmark documents for distribution-matching metrics. Parts 1 and 2 settle the question directly.
Why not just stop when the selector ensemble's variance gets large, the uncertainty-weighted rule? The clips a minimum-ensemble selects are, by construction, ones all members score highly, so their disagreement is suppressed exactly when hacking happens. A judge that took no part in selection is not subject to that selection effect. Ablation two tests this claim. If self-variance stopping performs as well, the monitor is unnecessary and we would say so.
Why not just tune one budget per genre on a validation set? Gao et al. tie the peak to judge quality, which plausibly varies within genres as well as between them. The per-prompt spread of $N^*$ measured in Part 2, together with the comparison against an oracle per-genre budget, decides whether per-prompt adaptation adds anything.
Why not just use a frontier multimodal LLM as the monitor? On CMI-Pref musicality, Gemini 3 Pro reached only 65.80% accuracy against CMI-RM's 78.20%. A monitor that weak would fire false alarms. It remains an ablation arm, not the default.
Decisive Checks
The first check is whether a single judge's gold reward peaks and declines within $N\le128$ in music. If it does not, the premise fails.
The second check is lead time. The monitor's toxicity rate must rise at least one budget level before gold reward peaks, measured as rank correlation between toxicity-rate increase and gold decline across prompts. Without lead time, the alarm is useless.
The third and final check is a blind pairwise listening test with listeners who have musical training, as in SongEval's annotation, on roughly 200 held-out prompts. It compares escalated search against the tuned fixed-$N$ worst-case ensemble at matched mean compute. We report win rate, mean samples per prompt, and the fraction of outputs rated in the bottom quartile, which is the analogue of the overdose-risk measure used in dose-finding simulations.
The idea survives only if it wins or ties on quality while lowering both compute and catastrophic outputs.
References
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
- Scaling Laws for Reward Model Overoptimization
- Phase I dose escalation study and pilot efficacy analysis of LXI-15029, a novel mTOR dual inhibitor, in Chinese subjects with advanced malignant solid tumors
- A new function for drug combination dose finding trials
- Reward Model Ensembles Help Mitigate Overoptimization
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- Estimation of median effective doses and 95% effective doses of oliceridine combined with propofol for day-case hysteroscopy under total intravenous anesthesia with preserved spontaneous breathing: an up-and-down sequential allocation trial
- MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques
- Technical Report on Shared Task in DialDoc21
- Assessment of Ensemble Learning to Predict Wheat Grain Yield Based on UAV-Multispectral Reflectance
- A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset