Text-to-music systems such as YuE, DiffRhythm, Suno and Udio are steered almost entirely by the words a user types. The automatic scores we would use to improve those words are known to disagree with human listeners. Fréchet Audio Distance, a statistic comparing the distribution of generated audio embeddings to real ones, can reward drums with no steady beat. The same statistical scores penalize jazz phrases that experts prize. Learned reward models compress their scores toward the middle of the scale and get exploited when optimized against. This proposal builds a prompt optimizer for music that uses the ideas of evolutionary prompt search, where a language model repeatedly rewrites and recombines candidate prompts and keeps the best. Its main job is to survive the fact that every available fitness signal is partly wrong.
The central observation is that a prompt optimizer is an adversary to its own judge. In music this adversary has an unusually cheap exploit. CMI-RewardBench showed that judges read the prompt when scoring: hiss is scored as a flaw under a neutral prompt and as intended texture under a "lo-fi" prompt. An optimizer that can rewrite the prompt can therefore rewrite the standard it is judged against. We propose to evolve a reusable prompt-rewriting instruction. Candidates are scored by an audio-language-model critic that always sees the user's original request rather than the rewritten prompt. Its written, dimension-by-dimension critiques drive the mutations. A second judge from a different model family, never used for selection, measures how much of the apparent gain is judge-specific. The deliverable is both a better music prompter and a measured account of how much search pressure turns into human-perceived quality versus exploitation of the judge.
Core Claim
Reflective prompt evolution can improve human-judged quality of black-box music generation over the user's raw request and over one-shot language-model prompt expansion. This holds only if two conditions are met. First, fitness must be computed against a fixed "intent anchor", the original brief. Second, mutation must be driven by the critic's natural-language diagnoses rather than its scalar score. Without the anchor, we predict the optimizer will add style framings (lo-fi, ambient, demo, live recording) that raise judge scores while humans, holding the original brief in mind, prefer the output less. Quantifying this gap is itself a contribution. The literature has documented perceptual-objective divergence only statically. It has not shown how divergence is amplified under optimization pressure on the prompt.
We define the hacking gap for a method as
$$\Delta = z_{\text{opt}} - z_{\text{human}},$$
where $z_{\text{opt}}$ is the method's improvement over the raw-brief baseline on the judge used for selection, and $z_{\text{human}}$ is its improvement in human pairwise preference. Both are standardized against the baseline's spread on held-out briefs. A method that truly improves music has $\Delta$ near zero. A method that games its judge has large positive $\Delta$.
Relation to Prior Work
The method comes from three lines of evolutionary prompt search. GEPA mutates instructions by having a language model read execution traces of failures and keeps a Pareto set across problem instances. It beat GRPO by about 6 points on average with up to 35× fewer rollouts. Its feedback is programmatic, from compilers and answer checkers, and it never faces a judge that can be manipulated through the prompt. CriSPO feeds multi-aspect critiques and suggestions into a prompt rewriter for summarization and question answering. We borrow its critique-then-rewrite structure but replace text references with a listening critic. APPO runs retain, align and expand steps for text-to-image with live human choices. We remove the human from the loop and confront the automated-judge problem it sidesteps.
The motivating evidence comes from the audio side. A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset and Improving Automatic Jazz Melody Generation by Transfer Learning Techniques show distribution metrics missing rhythmic incoherence and penalizing valued stylistic deviations. SongEval shows aesthetic quality is dissociated from fidelity. ViSQOL v3 notes that full-reference metrics break on generative synthesizers. These results rule out Fréchet distance or spectral error as fitness, and we use them only as diagnostic readouts.
The critic comes from MuseCritic. It trains an 8B audio-language model to write critiques on five aesthetic dimensions before scoring, which recovers score spread. MuseCritic uses its rewards for GRPO weight updates and Best-of-N reranking. It never feeds its critiques back into the prompt, which is what we do. LoopTTS is the closest in spirit: a Gemini judge diagnoses local prosody flaws and writes instructions, but these steer an audio refiner on one utterance rather than evolving a reusable prompt for a frozen generator.
The main constraint comes from CMI-RewardBench. It reports prompt-dependent musicality, high entanglement between its musicality and alignment heads (SRCC around 0.85), and vulnerability to reward hacking. Two design choices follow from it: anchoring on the original brief, and using a held-out judge from another model family. Pareto Prompt Optimization argues that dominance rankings are robust when training and test metrics differ. The music results above suggest objective rankings can invert human ones, so we test that robustness claim directly instead of assuming it. Tango 2 aligns text-to-audio by weight-level DPO on perturbed-caption preference pairs. It shows that complex prompts cause dropped or reordered events, but it is unavailable for closed generators.
Research Program
The system has four parts. A rewriter language model takes a user brief (text, possibly lyrics) and emits a generator prompt. The object being evolved is the rewriter's instruction, so the result transfers to new briefs at the cost of a single generation each. A frozen generator produces audio: YuE and DiffRhythm are open, and one commercial API is included if access permits. The optimizing critic is MuseCritic. For each clip it writes critiques on coherence, memorability, vocal naturalness, structural clarity and musicality and outputs per-dimension scores. It is shown the original brief, never the rewritten prompt. A held-out judge, CMI-RM, scores the same clips against the original brief and plays no role in selection.
Each generation of search samples a minibatch of training briefs and runs each surviving rewriter instruction. The system then collects the critic's critiques. A mutator model reads the critiques of the worst-scoring clips together with the prompts that produced them and proposes an edited instruction, as in GEPA. A merge step combines instructions from different Pareto members. Following GEPA, the Pareto set is kept per brief rather than per metric, because the file's evidence of entangled judge axes makes per-dimension Pareto fronts unreliable. Clips are 60 to 120 seconds, since CMI-RewardBench found longer segments improve judge accuracy. Every method receives the same fixed budget of generator calls.
The analysis arm catalogues what evolution changes. We tag each evolved instruction and prompt for added style framings not implied by the brief, removed constraints, and structural directives such as section markers or tempo. We then relate these edits to $\Delta$, which gives a concrete record of which prompt edits buy real quality and which buy only judge approval.
Baselines
The naive reference point sends the user's raw brief directly to the generator. All improvements and the standardization in $\Delta$ are measured against it.
The first practical competitor is one-shot prompt expansion. A strong language model rewrites each brief with a carefully hand-written music-prompting instruction, using the same rewriter model and a single call. To beat it, the evolved rewriter must win human pairwise preference on held-out briefs with a confidence interval excluding 50%.
The strongest existing inference-time competitor is Best-of-N reranking with a reward model, which CMI-RewardBench reports consistently helps. No reported result was found on Lacuna for the size of the Best-of-N gain in human preference. We run it with MuseCritic as the reranker at matched test-time generator calls: evolved-rewriter-with-Best-of-N against raw-brief-with-Best-of-N. We also run it at matched total calls, charging the optimization budget against Best-of-N amortized over the test briefs. The evolved rewriter beats Best-of-N if it wins in the first comparison and ties or wins in the second.
For judge quality we anchor to reported numbers. MuseCritic reaches 71.35% pairwise accuracy on Music Arena (MuseCritic). CMI-RM reaches 73.43% on Music Arena and 78.20% on CMI-Pref musicality (CMI-RewardBench). A roughly 25-30% error rate in either judge is the headroom an optimizer can exploit.
Three ablations isolate the key ingredients. The first is scalar-only evolution: the same search, but the mutator sees only scores and not critiques. This separates GEPA's claim that reading diagnoses matters from the effect of search itself. The second is unanchored evolution, in which the critic scores against the rewritten prompt. It isolates the anchor and is the configuration where we predict the largest $\Delta$. The third is objective-metric evolution, where fitness combines Fréchet Audio Distance against genre-matched references and CLAP text-audio similarity. It tests whether dominance over objective metrics actively selects against human preference, as the jazz and drum-loop evidence predicts. Every variant gets the same generator-call budget, the same rewriter and mutator models, and the same number of search generations.
Why not just…?
Why not just fine-tune the generator with GRPO or DPO on critic rewards? For Suno and Udio, both among the SongEval generators, weights are not available, so this is impossible. For open models it is possible, and MuseCritic did it for a 0.6B generator. It still faces the reward hacking CMI-RewardBench warns about, and GEPA's comparison suggests it needs far more rollouts than reflective prompt search. We will not claim superiority over weight tuning on open models. We include a small GRPO run on DiffRhythm, with rollouts matched to our search budget where feasible, and report both human preference and $\Delta$. If GRPO wins on open models, our contribution narrows to black-box generators, and we will say so.
Why not just use Best-of-N with MuseCritic at test time? This is the real threat. It needs no search, and if the raw brief's sampling spread already covers good outputs, it may match us. Best-of-N cannot fix a brief that systematically under-specifies structure, and its cost multiplies with every request for multi-minute songs. The matched-test-call and matched-total-call comparisons in the Baselines settle this directly. If evolved prompts add nothing over Best-of-N at matched total compute, the method fails on quality. The $\Delta$ analysis would remain a finding, and Best-of-N's own $\Delta$ is measured too.
Why not just ask an LLM to expand the prompt with rich musical vocabulary? It is cheap and is what practitioners already do. Tango 2 shows generators drop or scramble content as prompts grow more complex, so richer text is not automatically better. CriSPO found systematic critique-driven optimization beat manual prompts by 3-4 ROUGE points in text. Whether this carries over to music is exactly what the one-shot expansion baseline tests. If it ties the evolved rewriter, then listening feedback added nothing, and we would report that plainly.
Why not just optimize a weighted sum of objective and learned scores? Pareto Prompt Optimization found scalarized objectives collapse to a single compromise point in text. The more serious issue in music is that the objective components may rank artistically valid outputs lower. The objective-metric ablation measures this directly as human preference and $\Delta$ under that fitness.
Proposed Experiments and Decisive Checks
Briefs are drawn from the text and lyric instructions in CMI-Pref and the Music Arena pairs, split by brief into search and held-out sets. Human evaluation uses musically trained listeners in blind pairwise comparisons on held-out briefs. Listeners always see the original brief and are asked which clip better serves it musically. SongEval-style five-dimension ratings are collected on a subset to check per-dimension effects.
The first decisive check is quality. Anchored, critique-driven evolution must beat raw briefs and one-shot expansion in human preference on held-out briefs, and must beat Best-of-N under the matched-compute rules above. The second check is the anchor. Unanchored evolution should show a clearly larger $\Delta$ than anchored evolution, and more added style framings absent from the brief. If the two show no difference, the contextual-musicality exploit is not a practical threat and the anchor is unnecessary. The third check is judge independence. Gains on MuseCritic that do not appear on the held-out CMI-RM, or in humans, flag hacking, and we will report how often the two judges disagree about which candidate improved. The fourth check is critiques versus scores. Critique-driven mutation should reach a given held-out human preference in fewer generator calls than scalar-only mutation. If it does not, GEPA's central mechanism does not transfer to perceptual feedback. The fifth check is transfer. An instruction evolved on YuE is applied unchanged to DiffRhythm. The secure-code GA study (Discrete prompt optimization using genetic algorithm for secure Python code generation) saw transferred prompts fall below raw baselines, so a drop here would be informative rather than surprising.
References
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
- GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
- CriSPO: Multi-Aspect Critique-Suggestion-guided Automatic Prompt Optimization for Text Generation
- Preference-Guided Prompt Optimization for Text-to-Image Generation
- A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset
- Improving Automatic Jazz Melody Generation by Transfer Learning Techniques
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
- MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques
- Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
- Pareto Prompt Optimization
- Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization
- Discrete prompt optimization using genetic algorithm for secure Python code generation