Text-to-image diffusion models turn a written prompt into a picture by repeatedly denoising random noise. They still fail at simple spatial instructions. "A cup under a table" tends to come out as a cup on a table. "A dog behind a car" comes out with the depth order wrong. A plausible cause is that these models learn spatial relations from web captions, and those captions almost never state where things are: spatial prepositions appear in fewer than 0.2% of LAION-2B captions, according to What's "up" with vision-language models?. The model therefore falls back on which arrangements are typical for a given pair of objects. Static benchmarks hide this weakness, because their prompts mostly ask for typical arrangements of common object pairs. Those prompts sit close to the training distribution, and some have been used directly for reward tuning.
We propose to give the relation words in a prompt their own cross-attention pathway inside the image generator. Cross-attention is the step where image features query the prompt tokens. The new pathway would be trained on procedurally generated scenes in which object identity carries no information about the relation. We would evaluate it with a prompt generator that is resampled at every evaluation and has explicit knobs for how strongly a requested arrangement contradicts the usual one. The claim we test is narrow. Decoupling helps only because it lets the "where" pathway be trained under a distribution in which "what" is uninformative. Fine-tuning a shared pathway on the same data cannot get that benefit, because the shared projections keep absorbing object co-occurrence statistics.
Core Claim
In standard cross-attention, all prompt tokens are projected by the same key and value matrices and compete in a single softmax. A relation token such as "under" is a few low-weight tokens next to dense, strongly pretrained object tokens. It loses that competition, and whatever spatial signal it carries is entangled with object identity in the same projections.
We add a relation stream with its own keys and values. Its inputs are relation triples parsed from the prompt (subject slot, relation, object slot). Each slot is represented by a role embedding plus a pointer to the position of that entity token in the main prompt, not by the entity's text embedding. The layer output becomes
$$h = \mathrm{Attn}(Q, K_e, V_e) + \lambda\, \mathrm{Attn}(Q, K_r, V_r),$$
where $Q$ are the shared image queries, $K_e, V_e$ come from the frozen original text pathway, $K_r, V_r$ come from the new relation encoder, and $\lambda$ is a learned per-layer gate initialised at zero so the pretrained model is unchanged at the start of training.
During training, we sample the object pairs filling each relation uniformly and independently of how often that pair appears in that arrangement on the web. The relation stream therefore never sees a correlation between object identity and relation. We predict three things. First, the gap in relation accuracy between typical and counter-typical prompts will shrink under the relation stream. Second, the same procedurally generated data used to fine-tune a shared-attention model will improve typical prompts but leave most of that gap in place. Third, the advantage will be largest for relations that cannot be reduced to 2D boxes: depth order with occlusion, support and contact ("on", "under", "leaning against"), and containment.
Where This Sits
The architectural idea comes from three generative systems that use decoupled cross-attention to stop one conditioning source from overwhelming another. CustomNet puts object features plus camera viewpoint in one branch and background text in another, so the text cannot overwrite object geometry. MasterWeaver separates face identity from text and adds losses that keep edits working. RelationAdapter injects visual relation tokens into the FLUX transformer through dedicated key and value projections that share the backbone's queries. We reuse RelationAdapter's shared-query injection when we port the method to FLUX.
All three decouple a visual reference from text. None of them separates spatial syntax from entity semantics inside the text itself. All three are also evaluated only on static image sets.
AnyMS constrains the design. Its crop-and-merge step isolates each subject inside a bounding box, which stops attributes bleeding between subjects but cuts the cross-object links that occlusion and contact require. That is why our relation stream attends over the whole image rather than over cropped regions.
The motivation comes from two places. Kamath et al. explicitly suggest cross-attention architectures as a way to capture spatial relations, but they never build one. T2I-CompBench and T2I-CompBench++ report that SD3, DALL-E 3 and FLUX.1 all still fail on 3D depth arrangements.
The evaluation design borrows from contamination-resistant benchmarking. DyVal generates reasoning tasks on the fly from graphs with tunable depth. GPT-3.5-Turbo's arithmetic accuracy fell by 23% from the easiest to the hardest level, and reversing the order of the description cost GPT-4 13.6% on reachability. Spatial CAPTCHA procedurally renders 3D puzzles that frontier models solve at only 29–36%. Both evaluate understanding models and propose no fix. We carry their procedural design over to evaluating generation.
One finding cuts against us. Kamath et al. report that fine-tuning on spatial data can lower performance on controlled out-of-domain images. We read this as evidence that shared-pathway fine-tuning learns dataset-specific shortcuts. That is exactly the effect our identity-scrambled relation stream is meant to avoid, and our same-data baseline tests the reading directly.
Research Program
There are three components.
1. Procedural prompt generator. It samples scene graphs: objects as nodes, spatial relations as edges. The vocabulary spans four relation families: lateral (left/right), depth (in front/behind, with occlusion), support/contact, and containment. Graph depth runs from one relation to chains of four, mirroring DyVal's levels. Each edge also gets a typicality score, defined as how often that subject–relation–object pattern appears in a caption corpus relative to its reversal. This gives a prior-conflict knob that runs from "cup on table" to "table on cup." The generator also emits paraphrase-reversed pairs ("A behind B" / "B in front of A") that must yield the same scene, and mirrored lateral pairs in the spirit of 3DSRBench's FlipEval (3DSRBench).
2. Geometric verifier. It uses vision only: an open-vocabulary detector, a monocular depth estimator, and mask-adjacency checks for contact. It scores whether a generated image satisfies each edge. We use it for both training-data filtering and evaluation, and audit it against human labels on a stratified subset.
3. Relation stream. We train it on images generated by the base model from generator prompts and kept only when the verifier certifies them. This is GORS-style reward-based sample selection, extended to oversample certified counter-typical scenes. The base model is Stable Diffusion v2, which keeps the comparison to the published GORS numbers. We replicate on FLUX.1-dev using RelationAdapter-style injection.
Baselines
The naive reference is the unmodified base model on the same freshly sampled prompts. It sets the size of the typicality gap we are trying to close.
The strongest existing method on this question is GORS fine-tuning from T2I-CompBench, which fine-tunes the generator on its own reward-selected samples. The paper reports attribute binding (B-VQA) rising from 0.5065 for SDv2 to 0.6603 with GORS. No reported result was found on Lacuna for GORS or SDv2 on the 3D spatial category, so we will measure those ourselves. We run GORS on exactly our verifier-filtered, counter-typical-oversampled data, with the same number of images, gradient steps, and hyperparameter search budget as the relation stream. This baseline is also the ablation that isolates our key ingredient: the data is identical, and the only difference is whether relations get a separate pathway. Beating it means a statistically reliable reduction in the typical-versus-counter-typical accuracy gap at matched or better typical accuracy and prompt alignment (CLIP-T).
The layout baseline is a planner that maps the relation graph to boxes, fed into AnyMS. AnyMS reports 49.75 mIoU layout accuracy on its multi-subject suite. We expect it to be hard to beat on lateral relations. Our claim is limited to the depth, contact, and containment families, where boxes underdetermine the scene.
The reweighting baseline keeps joint attention but multiplies the attention logits of relation tokens by a tuned constant at inference. It tests whether the benefit comes from separating the softmax or from separate training.
The unscrambled ablation trains the relation stream with entity text embeddings in its inputs and natural co-occurrence pair sampling. If it matches the full method, decoupling helps through capacity rather than through the training distribution, and our stated mechanism is wrong.
Why not just…?
Why not just fine-tune the whole model on the same procedural data? This is the most serious threat. It is the GORS baseline above, run with identical data and compute. Kamath et al.'s finding that spatial fine-tuning learns shortcuts suggests it will close less of the counter-typical gap, but that is a prediction, not a result. If same-data GORS matches the relation stream on counter-typical prompts in the depth and contact families, we drop the architectural claim and report the procedural benchmark and training recipe as the contribution.
Why not just plan boxes with a language model and use layout-guided generation? For left/right this probably works as well or better, and we will say so. Boxes, however, do not specify occlusion order, contact, or containment. AnyMS's cropping also prevents objects from interacting across box boundaries, which is the open problem its own design creates. The per-family breakdown settles which approach wins where.
Why not just boost attention on the preposition tokens? It is free and training-free, which makes it a reasonable first try. But it only changes how loud the relation tokens are, not what their projections encode. If "under" is entangled with "cup-on-table" statistics, amplifying it should amplify the prior too. The reweighting baseline measures this directly, and a strong showing would mean the joint softmax was the whole problem.
Why not just wait for larger generators? T2I-CompBench++ reports that FLUX.1 and SD3 still fail on 3D depth. On the understanding side, Vision Language Models See What You Want but not What You See finds that scale buys almost nothing for perspective-taking. Scale is still a fair control, so we include FLUX.1-dev untouched as a larger reference point.
Decisive Checks
The primary metric is verifier-scored relation accuracy, reported per relation family and per typicality bin. It is evaluated on prompts freshly sampled at test time from seeds never used in training, at graph depths one through four.
The headline quantity is the prior gap: accuracy on typical edges minus accuracy on counter-typical edges. We report it alongside reversal consistency (the fraction of paraphrase-reversed pairs whose images both satisfy the relation) and flip consistency for lateral relations. Image quality and prompt alignment are tracked with CLIP-T and a blinded human preference study on 500 stratified prompts. The same human study audits the verifier, and if verifier-human agreement drops below 85% in any family we report that family on human labels only.
As a contamination signature, we also report each method's score on T2I-CompBench's static spatial prompts next to its score on matched procedural prompts with the same objects and relations. A method whose static score rises much more than its procedural score is fitting the benchmark rather than the relations.
The proposal succeeds if three things hold together. The relation stream must shrink the prior gap more than same-data GORS in the depth, contact, and containment families. Its accuracy must decay more slowly with graph depth. And the identity-scrambling ablation must account for most of that difference. If any of these fails, the corresponding paragraph above says which part of the claim falls.
References
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models
- MasterWeaver: Taming Editability and Face Identity for Personalized Text-to-Image Generation
- RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers
- AnyMS: Bottom-up Attention Decoupling for Layout-guided and Training-free Multi-subject Customization
- T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation
- T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Spatial CAPTCHA: Generatively Benchmarking Spatial Reasoning for Human-Machine Differentiation
- 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
- Vision Language Models See What You Want but not What You See