Spending Tired Listeners Where the Critic Lies: Human-Repaired Reward Models for Latent Music Evolution

Automatic music critics are now good enough to steer a generator, but not good enough to trust. A reward model, meaning a network trained to predict which of two clips human experts prefer, agrees with experts only about four times in five. When an optimizer pushes hard on such a critic, it finds clips the critic loves and people dislike. Humans can catch these failures, but a listener tires after a few dozen judgments and starts answering inconsistently. We propose an evolutionary search over a music generator's input space with three parts. The machine critic does almost all of the scoring. A small correction learned from human feedback repairs the critic where it is wrong. The few human judgments go only to one kind of question: "the critic says this evolved clip beats the clip it descended from; does it?" A model of how each listener's reliability decays over the session decides how much each answer counts and when to stop asking.

Put plainly, the human is not the fitness function, and the critic alone is not either. The human is an auditor who checks the specific places where search is exploiting the critic. Their answers become a patch on the critic, and that patch is trusted less as the listener tires. If this works, a session of roughly twenty short comparisons should produce results that beat both critic-only search and fully manual interactive evolution, as judged by independent listeners.

Core Claim

The search runs in latent space. A frozen generator $G$ maps a latent vector $z$ (the generator's input noise) to audio $x = G(z)$. A pretrained music critic $r$ scores the audio. The evolutionary algorithm optimizes the repaired fitness

$$f(z) = r(G(z)) + g(z),$$

where $g$ is a Gaussian-process correction over latent space. It starts at zero and is fit only to human answers. This additive form is borrowed directly from Preference-Based Reward Repair. That work repairs misspecified proxy rewards in control simulators by learning a human-derived term added to the proxy. It asks people to compare the current policy against a safe reference, and it needs far fewer queries than learning a reward from scratch. We carry that idea into a setting it never touched: a generative latent space, with audio judged by tired people.

The claim is that this transfer is non-trivial and that its value depends on two ingredients.

The first ingredient is ancestor-anchored queries. Each elite candidate is compared with the unoptimized sample it descended from in the search tree. This comparison isolates exactly what optimization added, and that is where reward hacking lives. A generic choice of "which of these K is best" mixes hacking with ordinary taste.

The second ingredient is a fatigue-aware likelihood. The probability that listener judgment $t$ prefers clip $a$ over clip $b$ is modeled as

$$P(a \succ b) = \sigma\!\left(\frac{u(a)-u(b)}{\tau_t}\right).$$

Here $u = r \circ G + g$ is the repaired utility, $\sigma$ is the logistic function, and $\tau_t$ is a noise temperature that grows over the session. We estimate $\tau_t$ online from hidden repeat trials, which are duplicated pairs the listener has already judged. This lets late, sloppy answers move $g$ less than early, careful ones. It also gives a principled stopping rule: stop when repeat-trial consistency falls below a threshold.

Why These Three Directions Meet Here

The motivating evidence comes from the metric-divergence literature.

CMI-RewardBench builds a music reward model that reaches 78.20% accuracy on its expert musicality preferences, while a frontier multimodal model reaches 65.80%. It warns that the reward model is open to hacking under direct optimization. Its "musicality" and "prompt alignment" outputs correlate at about 0.85 Spearman, so the critic cannot cleanly separate a musical clip from one that merely matches the prompt.

SongEval supplies a second, independent critic: an aesthetic predictor with 0.912 Pearson correlation to expert averages. It also documents strong entanglement among human aesthetic dimensions.

The FreeSound Loop benchmarking study shows how this plays out for drum loops. Distribution metrics reward drum-like spectra even when hits land off the beat.

Souaille et al.'s sound-design study is the closest audio precedent. It used surrogates to prune a 4,096-setting synthesizer space, which halved the interactive iterations needed. A behavioral surrogate beat the psychoacoustic formulas, but the pruning was offline and over six hand-made parameters.

The latent-search side supplies the method. Deep Interactive Evolution and Schrum et al.'s latent level design show that evolving a GAN's latent vector keeps every candidate valid, which spares users from rejecting garbage. In both, however, the human scores everything. SOE evolves diffusion noise seeds against two automatic critics with no human at all, so it inherits every blind spot of those critics.

MultiBO is the closest human-in-the-loop competitor. It runs Bayesian optimization over diffusion attention features from top-N-of-K choices. It needs 32–50 rounds of human feedback, and its authors name warm-starting from pretrained reward models as future work.

Our proposal differs from each of these. Unlike SOE, the critic is not trusted. Unlike DeepIE and MultiBO, the human is not the primary scorer. Unlike PBRR, the domain is a generative latent space, and the human's reliability is treated as non-stationary.

Research Program

We build two testbeds that differ in how expensive listening is.

The first testbed uses StyleGAN2 trained on the FreeSound Loop Dataset, with the setup from the FSLD benchmark. Its clips are two-second, 120 BPM one-bar drum loops, so a comparison takes seconds and fatigue can be studied over many trials. Search runs in the 512-dimensional latent space with CMA-ES, the covariance-adapting evolution strategy used for GAN latent search by Edwards et al..

The second testbed evolves initial noise seeds of a latent diffusion song generator from the SongEval model pool, such as DiffRhythm, in the manner of SOE. Listeners hear 20–30 second excerpts. Here each human judgment is expensive, so query allocation matters most.

In both testbeds the critic $r$ is CMI-RM's musicality output. The SongEval-trained predictor serves as a cheap second opinion. A generation triggers a human query only when the two critics disagree about whether an elite improved on its ancestor beyond a threshold. For drum loops we also use an onset-grid regularity check, since this is the failure that the FSLD study found distribution metrics miss.

Each human answer is a three-way judgment on an anchored pair: better, worse, or same. It updates $g$ through the likelihood above. The update reweights the current population immediately and penalizes the latent neighborhood of any hacked elite.

Before any real users take part, we run a controlled simulation. The SongEval predictor plays the "human" and CMI-RM plays the critic, so the divergence between them is real but measurable. Simulated fatigue is injected as a rising $\tau_t$ with lapses, and the schedule is fit to repeat-trial data from a short pilot. The simulation lets us sweep budget and fatigue rate cheaply. It also tells us whether the claimed gains appear before we spend participant time.

Baselines

The naive reference point is the unoptimized generator. We take one random latent sample per prompt for songs and per session for loops, at matched listening time. Any method that does not beat it with held-out listeners is not worth running. We also include a second weak reference: CMA-ES on the critic alone, with no human. It is the most direct test of whether hacking appears. If held-out listeners prefer its output to random samples at high optimization pressure, the problem we target is smaller than the literature suggests.

The strongest automatic competitor is best-of-N reranking with CMI-RM. Its paper reports that choosing the top of 10 samples improved quality as confirmed by human A/B tests. We run it with the same number of critic evaluations our search uses, which is in the thousands, not 10, so it gets the same compute. Beating it means a significantly higher win rate with held-out listeners in pairwise comparison. CMI-RM's reported accuracy is 78.20% on CMI-Pref musicality and 73.43% on Music Arena. We cite these as the level of critic agreement best-of-N inherits, not as a target we must exceed.

The strongest human-in-the-loop competitor is MultiBO's method, ported to our latent spaces. We use its multinomial-logit GP over top-N-of-K choices. In the strengthened variant its GP prior mean is set to the critic, which is the warm start its authors propose. It gets the same human query budget measured in listening seconds, the same generator, and its acquisition function tuned on the simulation. Its reported image result is a 70.82% win rate over fine-tuned baselines after 32–50 iterations, roughly 20 minutes. We found no reported result on Lacuna for MultiBO or any human-in-the-loop latent evolution on music. The comparison number must therefore come from our own run. Beating it means reaching equal held-out-listener quality with at most half its human queries, or higher quality at equal queries.

Pure interactive evolution in the style of DeepIE, with the human selecting every generation, is the fatigue-heavy reference. For songs we also include SOE's dual-critic seed evolution, using CMI-RM and the SongEval predictor as its two critics. It isolates whether two critics without a human already suffice.

Two ablations isolate our ingredients. The first keeps the additive repair and the human budget but replaces ancestor-anchored queries with top-N-of-K queries. The second keeps everything but fixes $\tau_t$ at a constant.

Why not just…?

Why not just rerank many samples with the best critic? This is a real threat for average-quality generation. CMI-RewardBench reports that best-of-10 reranking helps in human A/B tests. Reranking also applies weaker optimization pressure than evolutionary search, so it may hack less. Our position is that it cannot reach targets the prior rarely samples, and that its errors are capped by the critic's roughly 22% disagreement with experts. We test this in two ways. The compute-matched comparison above settles the open-ended task. A target-recreation task, where a listener tries to reproduce a held-out loop as in DeepIE, measures what reranking cannot reach. If reranking matches us on both, we will report that the human channel is unnecessary at these budgets.

Why not just use MultiBO with the critic as its prior mean? This is mathematically almost our fitness: a GP with prior mean $r$ is equivalent to a zero-mean residual added to $r$. So the additive repair is not our contribution. The question is whether which questions you ask and how much you trust late answers matter. The anchored-query ablation and the constant-noise ablation answer this directly. If warm-started MultiBO matches the full system, the honest conclusion is that PBRR-style anchoring does not transfer to latent audio search, and we will say so.

Why not just use critic disagreement and skip the human? An ensemble of the two critics could flag hacking on its own. But Souaille et al. found that objective acoustic measures missed the perceptual unpleasantness behavioral data caught. SongEval and CMI-RM may also share blind spots, since both are trained on outputs of similar generators. The SOE-style dual-critic baseline tests this. Our system should beat it specifically on the hacking rate defined below.

Why not fine-tune the generator offline on preference data? That aligns the model to the average annotator. MultiBO's result over DiffusionDPO suggests per-user steering wins on personal targets. We treat this as settled enough not to rerun. A DPO-tuned generator, if available, can serve as an additional starting prior.

Proposed Experiments

The decisive simulation. We measure, as a function of human query budget from 5 to 60, the "true" utility reached by each method, where truth is the SongEval predictor plus injected fatigue. Alongside it we track the hacking rate: the fraction of final elites the critic ranks above their ancestor but the truth ranks below. Critic-only CMA-ES should show a rising hacking rate under optimization pressure. If it does not, the premise fails.

The real study. About 30 participants each run the open-ended task ("best groove" or "best excerpt for this prompt") and the target-recreation task under each method in counterbalanced order. Every comparison carries hidden repeat trials so that $\tau_t$ can be measured rather than assumed.

Outcomes. Final outputs go to a separate panel of listeners who never ran a session. The panel judges them pairwise and on SongEval's five dimensions. Primary outcomes are panel win rate against the CMI-RM best-of-N and warm-started-MultiBO baselines, and human queries needed to reach a fixed panel score. Secondary outcomes are hacking rate and objective checks. For drum loops this is onset regularity. For songs we also use SteerMusic's CQT1-PCC, its melodic-retention metric, between elite and ancestor, to detect search that destroys melody while pleasing the critic.

Fatigue analysis. Fitted $\tau_t$ curves will show whether the reliability decay assumed by Wang et al. appears in short audio comparisons, and whether the repeat-trial stopping rule ends sessions before answers become worthless.

Falsification conditions. The claim fails if warm-started MultiBO matches the full system at equal budget, or if critic-only search shows no hacking on these generators.