Chain-Residue Leakage: Do Text-to-Image Models Bind Attributes They Have to Work Out?

Image generators often put the right colors on the wrong objects. Ask for "a red book and a yellow vase" and you may get a yellow book, a red vase, or two orange things. Almost all work on this attribute binding problem uses prompts where every attribute is written out right next to its noun. Real requests are often less direct. "The vase is the color of the book, and the book is the color of a ripe lime" requires the model to work out "green" before it can bind it. Such prompts also contain words the model should not draw, here "lime" and whatever color the book was first said to be. We propose a procedurally generated benchmark in which each attribute is reached through a controllable number of reasoning steps (hops), and every multi-hop prompt is paired with an explicit version of the same scene. The pairing lets us separate two failures: the model working out the wrong attribute, and the model working out the right one but letting words from the reasoning chain bleed into the picture. Because instances are generated fresh, and object-color pairs are sampled at chosen levels of real-world typicality, the same design also tests whether current binding scores depend on memorized combinations such as yellow bananas.

The central hypothesis is specific and falsifiable. Multi-hop prompts cause a failure we call chain-residue leakage: colors, textures, and objects that appear only in the intermediate steps of the reasoning chain show up in the image. We expect this failure to persist even after the chain has been resolved correctly in text, and we expect it to grow with chain depth. If this holds, both current fixes stop at the wrong point. Parser-guided attention steering binds whatever adjectives are on the surface of the prompt, and prompt rewriting resolves the logic but may leave residue words behind. If it does not hold, we will have shown cleanly that binding under reasoning reduces to resolving the text first. That is also worth knowing, because it tells the field where to spend its effort.

Why this gap, and what prior work does and does not settle

Three sets of evidence motivate the study. The first is about binding. Pre-trained text encoders treat prompts roughly as a bag of words, losing which adjective belongs to which noun (Ge et al.) and spreading context across padding tokens (Zhuang et al.). In diffusion models, the layers that link each word to image regions (cross-attention) produce overlapping regions for unrelated words, and color bleeds across the overlap (Zhou et al.; Hu et al.). A bag-of-words encoder has no reason to ignore "lime" just because it sits inside a clause of reasoning, which is why we predict chain-residue leakage.

The second is about reasoning over several steps. Quantifying Cross-Modality Memorization in Vision-Language Models trains vision-language models on synthetic personas and finds that they learn each hop but reach only 11% accuracy when two hops must be chained. DyVal builds reasoning problems as graphs of dependent steps and shows that accuracy falls with chain depth much faster than with chain width. Easier Painting Than Thinking, which introduces the T2I-CoReBench benchmark, shows that image generators handle reasoning-heavy prompts poorly and leak attributes more as scenes get denser. That paper is the closest existing work. Its 1,080 prompts are static and hand-written. It mixes several kinds of reasoning and does not vary the number of hops. Most importantly, it has no explicit twin for each prompt, so it cannot tell whether an image is wrong because of a reasoning error or because of leakage. We borrow its checklist-style grading and replace its prompt set with a generator that controls depth.

The third is about contamination. DyVal shows that models with strong static-benchmark scores (phi-1.5, WizardMath-13B) score near zero on freshly generated problems. Snoopy shows that few-shot accuracy tracks how often terms appear in the pretraining data. T2I-CompBench argues that its broad vocabulary of 2,470 nouns and 33 colors prevents memorization of common pairs, but it never tests this against typicality. MuSiQue supplies the method we reuse for multi-hop questions: build them bottom-up from single-hop pieces, then remove any question that can be answered without the intermediate steps. We apply this filter to prompts.

Some prior work constrains the design. RPG (Mastering Text-to-Image Diffusion) uses a multimodal language model to plan rectangular regions for each object. Rectangular regions struggle with irregular layouts, so we keep layouts simple to avoid confounding. SynGen (Linguistic Binding in Diffusion Models) depends on a syntactic parser and costs about 144% extra inference time. Its parser will link adjectives inside the reasoning chain to nouns, so we expect it to fail on residue in a particular way, and we will check whether it does.

Research Program

The core artifact is a prompt generator. Each scene has two to four renderable objects. Each object is assigned a color, texture, or material, drawn from the T2I-CompBench vocabulary so that the grading tools transfer. Each assignment is expressed through a small chain of definitions at depth $d \in \{0,1,2,3\}$, where depth 0 is the ordinary explicit prompt. In the closed-world arm, every fact is stated in the prompt, using in-context links such as "the vase matches the mug; the mug matches the bench; the bench is teal." No world knowledge is needed, so the arm is immune to contamination by construction, following the DyVal logic. In the open-world arm, links rely on real facts, such as "the color of the fruit that gives guacamole its base," so that parametric knowledge and possible memorization enter. Following DyVal, we also vary three secondary factors. We reverse the order in which the chain is presented, which DyVal found cost GPT-4 13.6% on its reachability task. We insert distractor links that name a color not used in the scene. And we vary residue load, the number of colors and objects that appear in the chain but not in the scene.

Each assignment of attribute to object is also sampled at a controlled level of typicality: typical (yellow banana), neutral (yellow mug), or counter-prior (blue banana). We estimate typicality from how often each pair co-occurs in a large public image-caption corpus. This yields frequency-inverted versions of T2I-CompBench-style pairs. Finally, every prompt is paired with an oracle depth-0 twin describing the identical scene, and a MuSiQue-style filter removes any prompt in which the target attribute can be guessed from surface cues alone. We check this by asking a text-only language model to predict the attribute with the chain deleted.

Grading assigns every object in every image one of four outcomes. It is correct. It shows neglect, meaning the attribute is missing. It shows swap leakage, meaning it carries another scene object's attribute. Or it shows residue leakage, meaning it carries an attribute that appears only in the chain. We also record whether a residue object, such as a lime, appears in the image at all, which corresponds to the "entity casting" failure. We grade with the disentangled question-answering approach used by T2I-CompBench, applied separately to each object, together with a multimodal-model checklist in the style of T2I-CoReBench. Both are calibrated against human labels on a stratified subset, and we report agreement.

The headline quantity is the derivation penalty. For each image generator and each depth, it is the drop in per-object correctness between a model given the multi-hop prompt and the same model given the oracle twin. We then split this penalty into two parts. The reasoning part is the portion explained by a resolver producing a wrong explicit prompt. The residue part is the portion that remains even when the resolved prompt is correct but the chain text is still present.

Baselines

The naive reference point is the image generator given the raw multi-hop prompt, with no resolution and no steering. We test Stable Diffusion v2, for which T2I-CompBench++ reports an attribute-binding score of 0.5065 (B-VQA), along with SDXL and open unified autoregressive models of the kind evaluated in T2I-CoReBench. Its oracle-twin counterpart gives the ceiling for each generator. All generators use the same seeds, the same sampler, and the same number of images per prompt.

The strongest existing methods fall into three families, and we run each in two conditions: on the raw prompt, and on a resolved prompt. The first is RPG, which plans regions with a multimodal model and diffuses each region separately. It reports a color-binding score of 0.8335 on T2I-CompBench, against 0.6369 for SDXL and 0.5750 for DALL-E 3, all figures from the RPG paper. It is the most direct competitor because its planner already reasons in steps before drawing. The second is SynGen, which reports 94.76% proper binding on the Attend-and-Excite set against 81.90% for the next-best method. It is the strongest training-free steering method. The third is GORS, which fine-tunes the generator on reward-selected images and raises Stable Diffusion v2 to 0.6603 B-VQA on T2I-CompBench++. For the open-world arm, the resolver in the resolved condition is either a single-pass language model or an interleaved retrieve-and-reason loop in the style of IRCoT. IRCoT reports 11–21 points of retrieval-recall gain and up to 15 points of question-answering F1 on multi-hop text benchmarks. No reported result on Lacuna exists for any of these methods on reasoning-derived attributes, so our depth-0 runs must first reproduce their published numbers on T2I-CompBench-matched typical pairs. This check keeps the comparison fair.

Every method uses the same resolver model, the same sampling budget, and default hyperparameters, with one small tuning sweep per method on a held-out depth-1 split. To count as beating a baseline, a method must show a statistically reliable reduction in residue leakage at depth 2 or more with no loss of depth-0 correctness. The key ablation isolates the residue hypothesis. We take a correct resolution, render once with the resolved prompt plus the chain text still present, and once with the chain text stripped. The difference between these two conditions is the residue effect.

Why not just…?

Why not just rewrite the prompt with a strong language model and then draw normally? This is the most serious alternative. T2I-CoReBench reports that rewriting helps weaker models, although stronger unified models gain little from it. Our design treats "resolve, strip, then render" as a full condition of the experiment. If it closes the derivation penalty to within noise at every depth, the residue hypothesis is false for that pipeline. The result would then be a clean decomposition showing that binding under reasoning is purely a text-side problem, which is itself useful. The live question is whether rewriting reliably strips the residue. Our ablation of "resolved with chain retained" against "resolved and stripped" measures this directly, and we will also measure how often free-form rewrites keep residue words.

Why not just use T2I-CoReBench? Its prompts are static, so they can be contaminated in the future. It mixes kinds of reasoning, does not control depth, typicality, or residue load, and has no explicit twins. Without twins, a wrong color cannot be attributed to a reasoning error rather than to leakage. We report our generators' scores on T2I-CoReBench alongside ours to show that the two benchmarks rank models differently.

Why not just audit existing benchmarks for co-occurrence frequency, as Snoopy does, instead of generating new prompts? A correlational audit is cheap, and we include it as a first step on T2I-CompBench. But frequency is tangled with noun concreteness and how easy an object is to render. The counter-prior arm varies typicality while holding objects and colors fixed, which a correlational audit cannot do.

Why not just scale the generator? The file's evidence argues against it. RPG, a planning method, outperforms larger commercial models on color binding. Cross-modal multi-hop accuracy stays at 11% across Gemma-3 sizes from 4B to 27B. We nonetheless report every trend across model sizes within each model family, so that a scaling explanation can be seen if it exists.

Proposed Experiments and Decisive Checks

The first check calibrates the benchmark. Depth-0 typical-pair scores should reproduce the published T2I-CompBench ordering of RPG, GORS, and base models. If they do not, the grader is suspect and we stop to fix it. The second check measures contamination. The drop from typical to counter-prior pairs at depth 0 shows how much of each method's reported binding comes from co-occurrence priors. We expect RPG's gain over SDXL to shrink on counter-prior pairs if its planner leans on familiar layouts and pairings.

The third check is decisive for the core claim. It plots residue leakage against depth and residue load, with and without the chain text, holding the resolution correct. Chain-residue leakage is real if leakage rises with residue load under correct resolution and disappears when the chain is stripped. It is refuted if the curve is flat. The fourth check separates reasoning from rendering. In the closed-world arm, reasoning failures should dominate for generators that read the chain themselves. In the open-world arm, interleaved retrieval should shrink the reasoning part of the penalty, and the remaining residue part tells us how far text-side fixes can go.

Finally, we fit per-object correctness as a function of depth and width, separately. If the DyVal pattern of depth mattering more than width carries over from text reasoning to image generation, that suggests the failure lies in the shared text-reasoning component rather than in the image renderer.