When a text-to-image model is asked for "a blue banana next to a red cup", it can fail in two different ways. It can smear red onto the banana because the two objects' regions overlap inside the model. Or it can keep the regions cleanly apart and still paint the banana yellow, because almost every banana it saw in training was yellow. A popular family of fixes enforces exclusive pixel assignment: each pixel of the image is owned by at most one object word in the prompt, so objects stop competing for the same image area. These fixes are reported to help a great deal. They are evaluated, though, on prompts and object pairings that the underlying model has very likely seen during web-scale pretraining. So nobody knows whether exclusivity teaches the model to bind an attribute to an object, or whether it only cleans up where things go and leaves what they look like to memorized priors.
This note proposes to separate those two effects. The test splits are built so that the model provably never saw the held-out attribute–object bindings. Leakage is then decomposed into three parts: missing objects, cross-object leakage (one object's attribute appears on another), and prior substitution (an object reverts to its most common training attribute). Our core prediction is that exclusive assignment nearly removes cross-object leakage on both seen and unseen bindings, but does little against prior substitution on unseen bindings, and may even make it worse. Once an object's region is isolated from the competing adjective, the region falls back on the object's training prior. If this holds, much of the reported compositional gain from attention segregation is a placement gain measured on contaminated benchmarks, and the open problem shifts from "where" to "what."
Background and the gap this fills
Exclusive assignment has a long history in vision. Examples include the single-parent constraint of Transforming Auto-encoders, the strictly partitioned canvas of Panoptic Segmentation, and the image-level softmax over region hypotheses in Interactive Full-Image Segmentation by Considering All Regions Jointly. In diffusion models (generators that produce an image by gradually removing noise from a random start), this principle reappears as losses or masks on cross-attention. Cross-attention is the map that tells each image location which prompt words to draw from.
These methods supply the techniques we will test. Attention Overlap Is Responsible for The Entity Missing Problem adds test-time losses (center-of-mass distance, overlap, divergence, compactness) that push object words' attention maps apart. Dynamic Prompt Learning optimizes per-step prompt tokens with a disjoint-attention loss. Bounded attention in Storybooth confines each subject's words to a region. SynGen confines an adjective's attention to its noun's region. All of these are evaluated on curated or static sets (COCO-Comp, MO-Set, T2I-CompBench). None asks whether the evaluated bindings were absent from training.
The motivating evidence comes from POLYOMINOGEN. It trains conditional diffusion models on a fully enumerable universe of colored polyominoes and holds out specific color–shape bindings. Zero-shot accuracy on the held-out bindings drops to 31.2% on the attribute split and 2.9% on the geometry split. The paper also introduces a Novel-Valid-Held-out metric that rejects near-copies of training images. Its testbed has only one object per image, so cross-object leakage cannot occur, and it never applies an exclusivity mechanism.
The contamination literature supplies the evaluation stance. DyVal regenerates reasoning problems procedurally and shows that models strong on static sets collapse on fresh ones. Snoopy shows that few-shot accuracy tracks how often test material appears in pretraining. Both work only on text.
What we add has three parts. We extend the POLYOMINOGEN-style controlled split to multi-object scenes, where leakage between objects actually exists. We decompose each error by cause using pixel-exact measurement. And we intervene with the exclusivity methods above, including an oracle version, to see which error component each one moves.
Core Claim
Exclusive pixel assignment addresses the spatial competition that Attention Overlap identified. It does not address binding against a memorized prior. Concretely, take a scene with objects $i=1..K$ and generated pixels $p$ inside object $i$'s region $R_i$. Let $a_i$ be the requested attribute, $a_j$ the attributes of the other objects, and $\pi(s_i)$ the most frequent training attribute of shape $s_i$. Each pixel can be scored as correct, cross-leaked, or prior-substituted:
$$ L_{\text{cross}} = \frac{1}{|R_i|}\sum_{p\in R_i}\mathbb{1}[\,c(p)\in\{a_j\}_{j\neq i}\,],\qquad L_{\text{prior}} = \frac{1}{|R_i|}\sum_{p\in R_i}\mathbb{1}[\,c(p)=\pi(s_i)\neq a_i\,], $$
where $c(p)$ is the attribute read off pixel $p$. The prediction is that exclusivity drives $L_{\text{cross}}$ toward zero on both seen and held-out bindings. On held-out bindings, however, $L_{\text{prior}}$ stays high and is roughly unchanged or larger with exclusivity than without. The claim would be falsified if exclusivity closed the held-out gap in the overall correct-and-novel rate as much as it closes the seen-binding gap.
Research Program
The first component is a procedural multi-object testbed. Scenes are rendered at 64×64 with one to four flat-shaded objects. Each object has a shape, color, and texture, and objects are placed with controlled overlap and occlusion. Captions are generated from a grammar, such as "a striped red L-tromino left of a blue square."
Because we render every training image, the training distribution is known exactly. That lets us define three held-out splits with certainty. The binding split contains color–shape pairs never seen anywhere. The co-occurrence split contains shape pairs never seen together, though each shape was seen alone. The prior-strength split contains bindings that were seen, but only rarely, against a strongly dominant prior color for that shape. Prior dominance is varied on a grid, so we can plot $L_{\text{prior}}$ against how lopsided the training prior was.
Test prompts are resampled with fresh seeds and layouts for every evaluation, in DyVal's spirit. Splits are defined at the level of attribute structure rather than surface templates, which addresses the concern that a procedural generator's own heuristics can be learned. Generated objects are recovered by connected components plus a small shape classifier trained only on renders. Their pixel colors give $c(p)$ directly, and the NVH novelty check is applied to each object crop.
The second component trains small text-conditioned diffusion models from scratch on this data. We train a U-Net and a diffusion transformer at three sizes each. This lets us ask whether the decomposition is architecture-specific, a question raised by Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation, and whether it changes with scale.
The third component is the open-world arm. For pretrained Stable Diffusion and FLUX.1 checkpoints, we build adjective–noun bindings and stratify them by co-occurrence frequency in LAION captions, using a Snoopy-style n-gram count. The strata run from common ("yellow banana") to essentially absent ("purple banana"). We then check whether the same pattern from the controlled testbed reappears as frequency decreases. This arm cannot certify zero exposure, which is why the controlled testbed carries the main claim. Region-level color is read inside segmenter masks, and T2I-CompBench's disentangled B-VQA is reported alongside it.
Baselines
The naive reference point is the unmodified diffusion model with standard sampling, evaluated on identical resampled prompts. It defines the baseline values of $L_{\text{cross}}$, $L_{\text{prior}}$, and the missing-object rate, separately for seen and held-out splits.
The strongest existing exclusivity method is the center-of-mass segregation loss of Attention Overlap. That paper reports a 24.1% improvement in VQA-based score over vanilla Stable Diffusion on COCO-Comp, and a 9% human-preference margin over Attend-and-Excite. It is the strongest competitor because it directly targets spatial competition, the mechanism we claim is only half the story.
We will also run the disjoint-attention loss of Dynamic Prompt Learning, Storybooth-style bounded attention, and SynGen's adjective-to-noun alignment, since SynGen is the method most likely to touch binding rather than placement. All test-time methods get the same sampler, number of denoising steps, guidance scale, and a matched hyperparameter search of equal size, tuned only on seen-binding validation prompts. Tuning on held-out prompts would contaminate the very split under test.
For training-time correction we include GORS fine-tuning from T2I-CompBench, which raised Stable Diffusion v2 attribute-binding B-VQA from 0.5065 to 0.6603. It is run with the same number of gradient updates as any fine-tuned variant.
No reported result was found on Lacuna for any exclusivity method on a provably held-out binding split. The relevant comparison therefore has to be generated here, and POLYOMINOGEN's 31.2% single-object zero-shot rate is the only anchor.
The key ablation is an oracle exclusivity arm. Ground-truth render masks hard-gate each object word's cross-attention to its own region. This gives perfect placement at no cost to tuning, and its held-out $L_{\text{prior}}$ is the cleanest test of the core claim. A second ablation combines the oracle mask with in-region upweighting of the adjective token. It tests whether, once placement is solved, attention intensity alone can override the prior.
Beating the field, for any proposed fix, means lowering held-out $L_{\text{prior}}$ beyond the oracle-mask arm without raising $L_{\text{cross}}$ or reducing NVH novelty.
Why not just…?
Why not just split T2I-CompBench prompts by web frequency and skip the synthetic testbed? This is a real alternative, and we run it as the open-world arm. However, caption n-gram counts cannot rule out that a "rare" binding appeared in images with different captions. A null result on rare prompts could also mean the frequency estimate is wrong rather than that exclusivity fails. The controlled testbed removes that ambiguity. If the two arms disagree, we report the disagreement and treat the controlled result as the mechanistic finding and the open-world result as its external check.
Why not just add a second object to POLYOMINOGEN? That is close to what the testbed is, and we credit it as such. A two-object version of POLYOMINOGEN alone would show that held-out multi-object scenes fail. It would not show why they fail. The contribution is the three-way pixel decomposition plus the intervention arms, especially the oracle mask, which attribute each failure to placement or to prior.
Why not just upweight the adjective token or raise guidance? Attention Overlap found that amplifying a weak token under overlap intensifies blending. Its finding concerns overlap, though, not priors. Under oracle masks, overlap is gone, so the oracle-plus-upweighting arm settles whether amplification suffices once placement is fixed. If it closes the held-out gap, the right conclusion is that exclusivity plus amplification is enough, and we would say so.
Why not just fine-tune on more combinations, or scale up? POLYOMINOGEN shows that a single example of a held-out binding lifts a pretrained model from 5.1% (scratch) to 54.8%. Data does help, but only for the bindings it covers. Our prior-strength grid measures how many examples are needed as a function of prior dominance. The size sweep tests whether larger models shrink $L_{\text{prior}}$ or merely memorize more.
Proposed Experiments and Decisive Checks
The decisive experiment compares oracle-mask exclusivity against the vanilla model on the binding split. If the oracle drives $L_{\text{cross}}$ near zero while held-out $L_{\text{prior}}$ stays within a few points of vanilla, and the correct-and-novel rate on held-out bindings rises far less than on seen bindings, the core claim stands. If instead the oracle closes the held-out gap, exclusivity really does induce binding, and the proposal's thesis is wrong.
The second check plots $L_{\text{prior}}$ against prior dominance. We expect a monotone rise that exclusivity shifts upward, not downward.
The third check runs the open-world frequency strata. We expect the segregation methods' B-VQA gain to shrink from common to rare bindings while the cross-object component stays suppressed.
The fourth check repeats all of the above on the diffusion-transformer models, to see whether unified multimodal attention behaves differently. All results will be reported with per-split confidence intervals over at least five training seeds and freshly resampled test sets.
References
- Learning to Parse Images
- Panoptic Segmentation
- Interactive Full Image Segmentation by Considering All Regions Jointly.
- Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!
- Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing
- Storybooth: Training-Free Multi-Subject Consistency for Improved Visual Storytelling
- Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment
- POLYOMINOGEN: A Controlled Testbed for Understanding Memorization and Compositional Generalization in Conditional Diffusion Models
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation
- T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation