Many recent image generators first write a plan and then draw from it. A language model lists the objects, their counts, their attributes and their bounding boxes, and an image model renders the result. This helps on hard prompts, but every prompt still gets the same planning effort and the same number of rendering attempts. We propose to rephrase each prompt a few ways that keep its meaning and to swap the example plans shown to the planner, then measure how much the resulting plans disagree with each other. Disagreement is cheap to compute because no image has been drawn yet. We test two uses for it: deciding how much extra rendering and repair effort a prompt deserves, and merging the plans into a constraint-level consensus that the renderer then follows.
The bet comes from text-only work. Language models are most sensitive to harmless rewording exactly where they are least likely to be right (On the Relation between Sensitivity and Accuracy in In-Context Learning measured Pearson correlations down to −0.40 between sensitivity and correctness). Nobody has asked whether the planner inside a plan-then-render generator is brittle in the same way, or whether that brittleness can direct test-time effort. If plan disagreement predicts which images will fail, compute can be spent where it matters. It would also give a pre-render failure detector and a way to measure the robustness of a whole class of methods that are currently evaluated with one wording per prompt.
Core Claim
In decoupled plan-then-render image generation, three kinds of input change are possible. The prompt can be paraphrased, the retrieved example plans can be swapped, and the planner can be sampled at nonzero temperature. Our claim is that the disagreement among plans produced under these changes predicts final-image failure better than two cheaper signals: the planner's own token confidence, and plain temperature resampling.
We further claim that allocating rendering and repair compute in proportion to this disagreement beats uniform allocation on accuracy at matched compute. It should also beat allocation by an image critic that only acts after rendering, because our signal is available before any pixels are drawn.
The measure is defined over parsed plans rather than raw text. Each plan $i$ is converted into a constraint set $C_i$: object classes with counts, attribute–object bindings, and pairwise spatial relations read off the boxes. For $K$ perturbed plans, disagreement is
$$d(x) = 1 - \frac{2}{K(K-1)}\sum_{i<j} J(C_i, C_j),$$
where $x$ is the user prompt and $J$ is Jaccard overlap between constraint sets. A value of $d(x)=0$ means every perturbation yields the same scene specification.
The consensus plan keeps the constraints that appear in a majority of the $C_i$. Box coordinates are averaged among the plans that agree on each relation. Averaging coordinates across several LLM layouts follows LLM Blueprint's interpolation step, but here it applies only within agreeing plans.
Research Program
Prior work supplies the parts. It also motivates the gap and constrains the design.
Planning stage. Two systems supply the planning stage we build on, and neither measures plan stability.
- T2I-R1 trains Janus-Pro to write a semantic plan before generating image tokens. It reports a 13% gain on T2I-CompBench, and its code is public (CaraJ7/T2I-R1).
- LLM Blueprint has a language model emit boxes and per-object prompts for a diffusion model. A diffusion model is a generator that turns random noise into an image by removing noise step by step.
Compute stage. Three methods spend extra test-time compute, each at a fixed amount per prompt.
- ImageGen-CoT samples several reasoning chains and several images per chain over static demonstrations.
- MILR runs a fixed 16 steps of gradient-based latent search.
- Iterative Refinement Improves Compositional Image Generation runs a critique-and-edit loop whose critic can stop early. It is the closest competitor, because its STOP action is already a form of adaptive compute, decided after rendering.
What we change. Our contribution is not a new planner or a new repair loop. It is a pre-render signal that decides how much of an existing repair loop each prompt receives, plus a consensus plan that may make the first render better.
Building the plan memory. We build a memory of verified plans for retrieval. On prompts disjoint from all test sets, drawn from T2I-CompBench training prompts and from captions with planning traces in FLUX-Reason-6M, we generate plans and render them. We keep only plans whose images pass deterministic detector-and-count checks. At test time we retrieve the nearest verified plans by embedding similarity, which is the nearest-neighbor demonstration selection of LM-BFF. Test prompts that nearly duplicate a memory entry are filtered out.
Retrieval plays two roles. It is an anchor that should make easy prompts converge. It is also a source of perturbation: Rethinking Hallucinations showed in text question answering that retrieval itself shifts with query wording. So retrieving with each paraphrase, rather than with the original prompt only, is part of the perturbation family. We expect this to expose instability that temperature sampling alone hides.
Allocating compute. The policy is deliberately simple. Prompts below a disagreement threshold get a single render from the consensus plan. Prompts above it escalate to the Iterative Refinement critique loop or to MILR latent search, with a round budget that grows with $d(x)$. The thresholds are tuned on a held-out split so that average compute matches a uniform baseline.
Counting compute honestly. We count all generated tokens: text tokens for the $K$ plans, image tokens for every render, and critic calls. We also report wall-clock time. If the $K$ planning calls turn out to cost a meaningful fraction of a render, that must show up on the compute axis.
Keeping plans short. More Thinking, Less Seeing? found that longer multimodal reasoning drifts away from grounding. We therefore cap plan length and use structured plans. We report whether consensus plans end up shorter than single plans, as that finding would predict.
Measuring brittleness. As a standalone measurement, we evaluate every system with eight meaning-preserving paraphrases per test prompt and report two quantities. The first is the all-paraphrase success rate, the fraction of prompts solved under every wording. The second is the variance of success across wordings. This applies the multiplicity framing of Rethinking Hallucinations to image generation.
Baselines
Naive reference. The plan-then-render model with one plan and one render per prompt, using T2I-R1's released Janus-Pro checkpoint and the original prompt wording. Every other method is reported as gain over this point. It also gives the paraphrase-variance floor that shows whether brittleness exists at all. We know of no measurement of this; no reported result was found on Lacuna for paraphrase robustness of any plan-then-render image generator.
Uniform parallel sampling with best-of-N. Sample N renders per prompt and pick the best with the same critic we use elsewhere, at the same total token budget as our method. It is the default way to spend compute. Iterative Refinement reports beating it by 16.9% in all-correct rate on ConceptMix at k=7, by 13.8% on T2I-CompBench 3D-spatial, and from 64.29% to 76.79% solve rate on Visual Jenga. So it should lose to both refinement and our method. If it does not on our backbone, the escalation operator is not working.
Strongest competitor: Iterative Refinement with its own critic-based stopping, at uniform budget. This is the method of Iterative Refinement Improves Compositional Image Generation, whose figures are quoted in the previous paragraph. It already adapts effort per prompt through STOP. We run it with the same multimodal critic, the same editor, the same maximum rounds and the same tuning budget for thresholds. We then compare accuracy-versus-compute curves. Beating it means a higher all-correct rate on ConceptMix k=7 and GIR-Bench-T2I at every matched compute level along the curve, with nonoverlapping bootstrap intervals, and not just at a single point.
MILR as a second escalation operator. MILR reports 0.95 overall on GenEval and 0.63 on WISE, and its code is public (spatigen/milr). We run it at its 16-step setting uniformly and under our allocation. Beating it on GenEval means matching 0.95 with fewer average latent steps, or exceeding it at equal steps.
ImageGen-CoT-style hybrid scaling. Several chains and several images per chain, run on our retrieved demonstrations instead of static ones. This separates the effect of retrieval from the effect of allocation.
Ablations isolating the key ingredient. Three ablations replace one piece at a time.
- Confidence instead of disagreement. Replace $d(x)$ with the planner's mean token log-probability. ProSA ties sensitivity to low decoding confidence, so confidence might be enough.
- Temperature only. Compute $d(x)$ from temperature resamples of the original prompt, with no paraphrases and no exemplar swaps.
- Static exemplars. Fix the exemplars instead of retrieving them.
The proposal's central ingredient holds only if the full perturbation family beats all three.
Why not just…?
Why not just let the refinement critic decide when to stop? This is the most serious threat. The critic looks at the actual image, while our signal only guesses. Our argument is cost: the critic has to see at least one render per prompt before it can decide anything, while plan disagreement is available before any render. The matched-compute curves settle this directly. If critic stopping matches our allocation at equal tokens, the allocation claim fails. We would then report only what survives: the consensus plan's first-render gain and the brittleness measurements.
Why not just sample the planner several times at higher temperature (self-consistency) instead of paraphrasing? Temperature resampling explores only the planner's uncertainty under one wording and one retrieval result. SENSEL found perturbation disagreement beat entropy and max-probability baselines for abstention, with +4.1 AUC on GPT-J and +3.0 from reordering alone. That evidence comes from text classification, though, not layout planning. The temperature-only ablation tests whether paraphrase and retrieval perturbations add predictive power, measured by failure-prediction AUROC.
Why not just canonicalize the prompt with one LLM rewrite? A single rewrite picks one point among many possible wordings. Rethinking Hallucinations shows that such rewrites can land in a different error region rather than a safer one. Black-box robustness optimization like PSM needs search loops and was built for prompt-extraction defense, not composition. We include a single-rewrite baseline. If it matches the consensus plan's all-paraphrase success rate, the consensus half of the proposal is unnecessary.
Why not just add more denoising steps? Iterative Refinement reports that more denoising steps improve alignment but stay inadequate for scenes with many constraints. Factuality Matters locates the bottleneck in planning, not rendering. We report a steps-scaled diffusion variant for completeness and expect it to plateau early.
Proposed Experiments
We use four benchmarks.
- GIR-Bench-T2I (GIR-Bench) is the primary test, because detectors and OCR score it deterministically, which limits reward hacking by the critic. Its best reported numerical-reasoning score is only 0.362.
- ConceptMix at k from 3 to 7.
- T2I-CompBench counting and spatial categories (T2I-CompBench).
- GenEval, for comparison with MILR.
Planners are the T2I-R1 Janus-Pro model and an LLM Blueprint-style planner driving a diffusion renderer. That lets us check that the signal is not specific to one architecture.
First decisive check: does disagreement predict failure, before any compute is spent on allocation? We generate one render per prompt and compute the AUROC of $d(x)$ for predicting failure. We compare it against token confidence and temperature-only disagreement. We commit in advance to abandoning the allocation claim if the full signal does not beat confidence-based prediction on GIR-Bench-T2I and ConceptMix k=7.
Second decisive check: matched-compute accuracy curves. We compare our allocation against uniform best-of-N, uniform Iterative Refinement with critic stopping, and uniform MILR, across five budget levels.
Third check: consensus plans and brittleness. We measure whether consensus plans raise the all-paraphrase success rate and lower success variance across wordings relative to single plans and single rewrites.
Fourth check: retrieval stability. We measure whether retrieval reduces disagreement on prompts that turn out easy without lowering disagreement on prompts that fail. Retrieval anchoring that silences the warning signal on hard prompts would be a failure mode worth reporting even if accuracy rises.
References
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts
- ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoning
- MILR: Improving Multimodal Image Generation via Test-Time Latent Reasoning
- Iterative Refinement Improves Compositional Image Generation
- FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark
- Making Pre-trained Language Models Better Few-shot Learners
- Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization
- Factuality Matters: When Image Generation and Editing Meet Structured Visuals
- GIR-Bench: Versatile Benchmark for Generating Images with Reasoning
- T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation