Lock What Is Right: Contrastive Label Verification for Text-to-Image Self-Correction

Text-to-image models still fail on prompts with several objects, attributes, and spatial relations. A popular fix is to let a vision-language model look at the output, say what is wrong, and ask an editor to repair it. Work on language models suggests caution here. When the same kind of model both generates and grades, the grader is often the weakest link. It rejects correct answers, "fixes" parts that were fine, and in the end often does worse than drawing fresh samples and checking them with a reliable external checker. This proposal takes that lesson literally for images. Before building a correction loop, we measure how often the checker is wrong on each kind of constraint. We then build a checker whose errors are rare and whose uncertainty is visible. Finally, we allow the loop to touch only what the checker confidently says is broken, and we freeze the image regions it confidently says are correct.

The checker borrows an idea from the literature on semantic labels: a class is scored by matching the input against natural-language descriptions of the class, instead of by an arbitrary index or a single yes/no question. Each atomic requirement in a prompt, such as "the mug is yellow" or "the cube is behind the plate," is checked by comparing the relevant image region against a description of that requirement and against descriptions of its most confusable alternatives ("orange," "in front of"). The result is a margin with an uncertainty estimate, not a yes-token probability that leans toward agreement. Put simply, correction loops fail because their graders are unreliable, and a grader built from contrasts between label descriptions, combined with a loop that never edits confidently-correct content, should beat both the naive loop and plain best-of-N sampling on the hardest multi-constraint prompts.

Core Claim

In compositional image generation, whether iterative correction helps is set mainly by the verifier's per-constraint error rates, especially false negatives on constraints that are already satisfied. How the loop is designed matters less. We claim three things. First, per-constraint false negatives of vision-language self-critics are large enough to explain the "drift" that correction loops report. Second, a hybrid verifier cuts those errors. It uses geometry from detected boxes for counts and 2D relations, and contrastive matching of label descriptions for attributes and depth relations. Third, a loop that repairs only confidently-failed constraints, while masking confidently-satisfied regions, beats compute-matched best-of-N under the same verifier, with the advantage growing as the number of constraints per prompt grows.

This is motivated by On the self-verification limitations of large language models on reasoning and planning tasks. That paper shows GPT-4 self-critique dropping Graph Coloring accuracy from 16% to 2%, with a 97.09% false-negative rate on Mystery Blocksworld, while sampling plus a sound verifier reaches 44% on Graph Coloring. Image generation has no sound verifier. The open question is therefore how close to sound one can get, per constraint type, and what the remaining unsoundness costs.

Relation to Prior Work

Three test-time systems already close the loop on compositional generation. Iterative Refinement Improves Compositional Image Generation uses a vision-language critic that chooses to continue, backtrack, restart, or stop, and sends text edits to an image editor. R³-Refiner trains the reflector with reinforcement learning and names the gap between noticing an error and producing an edit that fixes it. AlphaGRPO shows that multimodal models judge images better when prompts are broken into atomic yes/no rubrics. All three score constraints with a generative model's own verbal or yes-token judgment. None of them reports per-constraint false-negative rates, and all edit the whole image through a text instruction, so correct content is exposed to change. We differ in two ways: in how each constraint is scored (contrastive descriptions plus geometry, with abstention), and in what the repair may touch (only failed regions).

Open-loop planners supply our layout step but not our loop. LayoutGPT, StruVis, and RePrompt produce layouts or reasoned prompts once and never check whether the generator followed them. We reuse an LLM layout plan as the reference that repairs are checked against. Correcting Diffusion Generation Through Resampling already uses an object detector mid-generation to cull and clone candidate trajectories. That is the closest evidence that detector-based checking works, but it scores object presence only and resamples whole trajectories rather than repairing parts.

The verifier's design comes from the semantic-label literature. Adaptive Concept Discovery for Interpretable Few-Shot Text Classification routes decisions through human-readable concept descriptions. Insight finds hierarchical, nameable concepts inside vision-language encoders. We reuse the description-matching idea and add explicit contrast sets. That literature's known weaknesses also constrain us. Scores are sensitive to how descriptions are phrased, and close categories overlap in embedding space. We handle this by ensembling paraphrases and treating disagreement across paraphrases as uncertainty that triggers abstention.

Training-time fixes such as IPR and GORS in T2I-CompBench address the same failures offline. They are complementary rather than competing, and IPR's reported trade-off between fidelity and accuracy is one reason to prefer local, masked repair.

Research Program

The first phase is a verifier audit. We sample images from FLUX and SDXL on GenEval, T2I-CompBench++, and ConceptMix prompts, and split each prompt into atomic constraints of four kinds: object presence, count, attribute, and relation (2D or 3D). Each constraint is labeled by human annotators, and we add the expert annotations in R³-Bench. We then measure false-positive and false-negative rates per constraint type for four candidate verifiers: a holistic vision-language critic, AlphaGRPO-style decomposed yes/no questions, GenEval's detector-based checks, and our hybrid.

The second phase builds the hybrid verifier. Presence, count, and 2D relations are decided geometrically from open-vocabulary detector boxes, which is the closest thing to a sound check available. Attributes and 3D relations are decided by cropping the relevant region and scoring it with a dual image-text encoder against a small set of paraphrased descriptions of the target label. The same crop is scored against descriptions of its confusable alternatives, which are drawn from a fixed confusion table: neighboring colors, the opposite predicate, and adjacent counts. A constraint receives a verdict only when the target-versus-alternative margin clears a threshold and the paraphrases agree. Thresholds are calibrated per constraint type on held-out human labels. Everything else is marked "unknown."

The third phase is the lock-and-repair loop. An LLM turns the prompt into a layout of boxes. After generation, every constraint that is confidently satisfied has its region masked and frozen. Each confidently-failed constraint is repaired by inpainting inside its planned box, or, for a relation, by re-placing one object's box and inpainting it there. Unknown constraints are left untouched. The loop stops when no confidently-failed constraints remain or when the compute budget is spent.

The key quantity we track is drift: the fraction of constraints that were correct before a round and broken after it.

Baselines

The naive reference is a single sample from the base generator, FLUX or SDXL, with no correction. The file supplies anchor numbers. GenEval reports SD-XL at 55% overall and strong models near 15% on position and near 35% on attribute binding. RePrompt reports FLUX at 0.66 overall on GenEval, rising to 0.76 with its reasoning reprompter. RePrompt is also a baseline: it is the strongest open-loop method reported in the file.

The strongest competitor on this question is compute-matched best-of-N using our hybrid verifier to pick the winner. This is the direct analogue of the sampling-plus-sound-verifier result that beat self-critique in Kambhampati et al., and it is the baseline most likely to erase our gains. We match total denoising steps, with inpainting steps counted at their true cost, and give both methods identical verifiers, prompts, and seeds. We also run best-of-N with GenEval's own detector as the selector.

The strongest existing closed-loop method is Iterative Refinement. It reports a 16.9% improvement in all-correct rate on ConceptMix at k=7 over compute-matched parallel sampling, and 13.8% on T2I-CompBench 3D-Spatial. No absolute all-correct value for it on ConceptMix was found on Lacuna. We will re-run it with the same base generator and compute budget. For R³-Refiner we found only relative gains (+12.0% verdict, +9.0% rectification on R³-Bench), and we compare on R³-Bench where feasible.

Two ablations isolate the key ingredients. The first swaps our verifier for AlphaGRPO-style yes/no questions while keeping the lock-and-repair loop. The second keeps our verifier but removes locking, so repairs regenerate the full image through an editor. Beating the field means a higher all-correct rate than compute-matched best-of-N under the same verifier on ConceptMix at k≥5 and on T2I-CompBench++ 3D-spatial and numeracy, with lower drift than Iterative Refinement. Because optimizing against one checker inflates scores measured by the same checker, all headline numbers use a different detector from the one inside the loop, plus human judgment of each constraint.

Why not just…?

Why not just sample N images and keep the best one under a good detector? This is the real threat, and the Kambhampati results say it often wins. Our argument is arithmetic. If each of k constraints holds independently with probability p, a full sample succeeds with probability roughly p^k. Repair instead fixes failures one at a time, so the gap should widen with k. Iterative Refinement's gain over parallel sampling at ConceptMix k=7 is consistent with this. If our matched comparison shows no advantage at high k, the paper's contribution becomes the verifier audit, and we will report it that way.

Why not just ask the vision-language model decomposed yes/no questions, as AlphaGRPO does? Decomposition helps, but a single yes-token probability still inherits the model's tendency to agree. It also gives no way to distinguish "orange, not red" from "unsure." The verifier-swap ablation tests this directly. If the yes/no verifier matches ours on false negatives and on the loop's final accuracy, the contrastive-label component is unnecessary.

Why not just fine-tune the generator on its corrected failures, like IPR or GORS? Training changes the base rates, but it does not remove the need for test-time checking on unseen combinations. It also brings the fidelity loss IPR reports. We treat a GORS-tuned generator as an alternative base model and test whether lock-and-repair still adds gains on top of it.

Why not just use a bigger generator? GenEval's scaling study shows gains in presence of single objects but poor position and binding even at scale. We will nonetheless report every result on both SDXL and FLUX to show whether the gap narrows as the base model gets stronger.

Decisive Checks

Four checks decide the proposal. The first is whether the audit shows substantial false negatives on satisfied constraints for the holistic and yes/no critics, and whether those errors predict per-round drift in the Iterative Refinement loop. If they do not, the central causal story is wrong. The second is whether the hybrid verifier's calibrated false-negative rate is clearly lower per constraint type on held-out human labels, with the abstention rate reported alongside. The third is whether, at matched compute, lock-and-repair's all-correct advantage over verifier-matched best-of-N grows with k on ConceptMix. The fourth is whether the no-locking ablation reproduces drift and loses most of the gain, which would show that preserving correct content, and not repeated trying, is doing the work. Throughout, aesthetic scores and human preference are tracked to catch any fidelity cost of repeated inpainting.