Casual users type short requests like "a couch below a vase" into image generators. Those generators work best with long, carefully worded prompts, so systems now put a small language model in front of the generator to rewrite the user's request. The best of these rewriters are trained by reinforcement learning (RL): sample a rewrite, generate an image, score it with a judge, and update the rewriter. Each rewrite therefore costs a full image generation plus a call to a large vision-language model used as the judge. That judge is itself unreliable: it hallucinates, prefers longer text, and gives most images high scores. The rewriter learns to please the judge, which produces the "over-reasoning" failure that RePrompt reports, where the rewriter adds objects and attributes the user never asked for.
We propose dropping the external judge. The reward comes from the image generator itself, and it is anchored to the user's original words, not to the rewrite. Generate the image from the rewritten prompt, add noise to it, and measure how much better the generator's denoising network can remove that noise when told the user's short query than when told nothing. This is a pointwise mutual-information estimate. It is cheap: a handful of forward passes and no billion-parameter judge. It cannot be inflated by verbosity, because the rewrite's text never enters the score. A rewrite earns reward only if it makes the user's request more legible in the image.
Core Claim
An intent-anchored, self-scored reward lets a prompt rewriter be trained with critic-free RL at a fraction of the per-rollout cost of judge-based pipelines. It should match their compositional accuracy while producing fewer unrequested objects. Let $x$ be the user query, $p$ the rewrite sampled from the policy, $y$ the image generated from $p$, $z_t$ the image's latent noised to level $t$ with Gaussian noise $\epsilon$, and $\epsilon_\theta$ the frozen denoiser. The reward is
$$ r(x,p) \;=\; \frac{1}{K}\sum_{k=1}^{K}\Big[\,\lVert \epsilon_k - \epsilon_\theta(z_{t_k}, t_k, \varnothing)\rVert^2 \;-\; \lVert \epsilon_k - \epsilon_\theta(z_{t_k}, t_k, x)\rVert^2\,\Big], $$
averaged over $K$ sampled noise levels $t_k$. The first term is the denoising error with no text, and the second is the error when conditioned on the user query $x$. This is the denoising-based mutual-information estimator that Information Theoretic Text-to-Image Alignment (MI-TUNE) uses to rank candidate images. MI-TUNE then fine-tunes the generator on the top-ranked images, so it changes the backbone and does nothing for the user's underspecified query. We keep the generator frozen and repurpose its estimator as an RL reward for the rewriter. Our one change is to condition on $x$ instead of $p$, and that change carries the claim. Conditioning on $p$ would reward rewrites that are easy to render, regardless of whether they preserve what the user asked for.
Research Program
The policy is a mid-sized instruction-tuned language model. It is first supervised on rewrite pairs, using the DiffusionDB bootstrapping recipe from BeautifulPrompt. It is then trained with Group Relative Policy Optimization (GRPO), which scores several rewrites of the same query and normalizes rewards within that group, removing the separate value network. This choice addresses the four-model memory cost that P-TA describes for PPO. With an intrinsic reward, the reward model is also gone: only the policy, the frozen reference policy, and the frozen generator remain in memory.
Generation remains the dominant rollout cost, a bottleneck Is Reinforcement Learning (Not) for Natural Language Processing identifies for any generation-in-the-loop RL. We will therefore study how few sampling steps the reward tolerates. The question is whether ranking rewrites within a group needs a finished image, or whether a coarse low-step image suffices. DPO-Diff motivates this question: it shows that image properties can be estimated from one or two intermediate denoising steps rather than a 50-step unroll.
The work proceeds in three stages. First, we validate the reward offline. On T2I-CompBench and GenEval prompts we generate several rewrites per query and check whether the reward's ranking of the resulting images agrees with the GenEval detector verdicts. We compare that agreement with the agreement achieved by an open vision-language judge and by GPT-4V. Second, we train rewriters under matched compute. Third, we measure cost per rollout and the rate at which the rewriter's images contain unrequested content.
We train on SD3 and PixArt-Σ and test transfer to FLUX.1, the backbones RePrompt uses. Two risks are acknowledged up front. Adapting the estimator from MI-TUNE's diffusion setting to the flow-matching objective of SD3 and FLUX is a step we must verify. And the reward inherits the generator's text-encoder blind spots, which DPO-Diff documents for CLIP. For that reason, every success claim is made on external metrics the reward never sees.
Baselines
The naive reference point is feeding the raw user query to the generator unchanged. Next comes zero-shot expansion by the same base language model with a few-shot instruction, matching the "LLM enhancement" baseline that RePrompt's position gains are measured against. A gain over this baseline shows that training helps at all.
The strongest existing method is RePrompt, which trains a reasoning-then-prompt policy with GRPO. Its reward combines ImageReward, GPT-4V alignment judgments, and structure and length terms. It reports that overall GenEval on FLUX rises from 0.66 to 0.76, with relative gains of +77.1% on the position category. We run it from the released DKI_LLM codebase on the same training queries. Because a GPT-4V reward is impractical to re-run at scale, we also train a RePrompt-style variant with an open vision-language judge under our exact GPU-hour budget. No training-cost figure for RePrompt was found on Lacuna, so we will measure both pipelines ourselves.
Beating RePrompt means at least matching 0.76 overall GenEval on FLUX while lowering the unrequested-object rate, at a lower per-rollout cost. Matching accuracy at lower cost alone counts as a partial win. BeautifulPrompt, whose PPO rewriter reports PickScore 20.84 and Aesthetic Score 6.52, is included as the aesthetic-reward reference.
The key ablation trains an identical policy with the reward conditioned on the rewrite $p$ instead of the query $x$. If intent anchoring matters, this variant should score similarly on image–rewrite agreement but drift more from the user's request. Two further ablations vary the number of sampling steps used to produce rollout images and the number of noise levels $K$.
Why not just…?
Why not just rerank at test time with the same score instead of training a rewriter? Generate N rewrites, render all of them, and keep the one with the highest reward. This could match our accuracy, since it is how MI-TUNE curates data. The cost is N full generations per user request at deployment, the same latency problem RATTPO faces with its 100–160 search iterations. We include best-of-N with N up to 16 and report accuracy against inference cost. If best-of-4 equals the trained policy, the honest conclusion is that the reward matters more than the training, and we will say so.
Why not just replace GPT-4V with a cheap object detector or VQA reward? This is the most serious threat. A detector reward is also cheap and has no verbosity bias. GenEval's detector agrees with humans 83% of the time, versus 69.1% reported for MI-TUNE's estimator. However, a detector reward trained on GenEval-style checks optimizes the test metric directly, and it covers only the detector's object vocabulary. We train a GRPO rewriter on a detector reward with matched compute and evaluate both rewriters on metrics the other did not train on: T2I-CompBench's B-VQA, UniDet spatial and numeracy, and human preference. If the detector-trained rewriter wins on held-out metrics, the intrinsic reward is not justified.
Why not just use offline preference optimization instead of online RL? VidPrompter's hallucination-aware preference optimization (Hallucination-Aware Prompt Optimization for Text-to-Video Synthesis) avoids rollouts entirely. Its pairs are static, though, and MAPO argues that good prompts are specific to the target model. Our reward makes a middle path cheap: rank rewrites from the supervised policy by $r(x,p)$ once, then train with DPO on the resulting pairs. We run this as a baseline at matched generation count. If it reaches GRPO's accuracy, we adopt it as the recommended recipe, because the reward is the contribution, not the optimizer.
Proposed Experiments
Reward validity. Before any training, the reward must rank rewrites correctly. On GenEval queries, we measure pairwise agreement between the reward's ordering of images and the detector's correct/incorrect verdicts. We compare this with the agreement of an open vision-language judge, under the expectation set by MLLM-as-a-Judge, which reports only 0.490 Pearson agreement with humans for GPT-4V scoring. We also report the correlation between each reward and rewrite length. Ours should be near zero by construction, while judge rewards should rise with length if the reported verbosity bias holds here.
Main comparison. We evaluate on GenEval overall and per category, T2I-CompBench++ categories, and PickScore and aesthetic score on DiffusionDB queries, so that semantic gains are not bought with visual quality. For hallucination, we define an intent-drift rate: the fraction of generated images in which GenEval's detector finds an object class that is absent from the user query. We also count how many object nouns appear in the rewrite but not the query.
Decisive checks. The proposal succeeds only if the intent-anchored policy (a) matches RePrompt's 0.76 GenEval on FLUX within noise, (b) shows a lower intent-drift rate than both RePrompt and the $p$-conditioned ablation, and (c) costs measurably less per rollout in GPU-seconds and peak memory. It fails if the gap between the reward's ranking and detector verdicts widens during training, which would mean the policy has learned to exploit the generator's own blind spots. We track this gap at every checkpoint, as the self-scoring analogue of reward hacking.
References
- RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learning
- Information Theoretic Text-to-Image Alignment
- BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis
- P-TA: Using Proximal Policy Optimization to Enhance Tabular Data Augmentation via Large Language Models
- Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
- On Discrete Prompt Optimization for Diffusion Models
- Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models
- Hallucination-Aware Prompt Optimization for Text-to-Video Synthesis
- MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark