Variable-Depth Part Trees for Nested Attribute Binding in Diffusion

Text-to-image models often put attributes on the wrong object. Ask for "a red book and a yellow vase" and you may get a yellow book. The best fixes today handle one link at a time: an adjective and the noun it describes. Real prompts nest. In "a man in a green jacket whose zipper has a gold pull-tab," the word "gold" belongs to a pull-tab, which belongs to a zipper, which belongs to a jacket, which belongs to a man. We propose to treat this nesting as a chain of hops, much like a multi-hop question such as "in what city was the director of Inception born?", where each step depends on resolving the one before. The method writes the prompt as a part-whole tree. It then guides the image generator so each part's region stays inside its parent's region and apart from its siblings. The depth of each branch is set by how much image area the parent actually receives, not by a fixed number of levels.

The pitch in two sentences: current binding methods are flat, so their errors should compound with nesting depth, the same way linear retrieval chains compound errors across hops. A tree of containment constraints, with each branch cut off at the depth the canvas can resolve, should keep binding accuracy roughly level as prompts get deeper. The note also contributes a controlled benchmark that builds deep binding prompts from single-hop ones. It borrows the construction MuSiQue used to stop multi-hop QA models from taking shortcuts.

Core Claim

A diffusion model is a generator that turns random noise into an image by removing noise step by step. Inside it, cross-attention maps record which image locations each prompt word influences. Leakage happens when a modifier's map spreads onto another entity's region (Zhou et al.; Hu et al.). SynGen (Linguistic Binding in Diffusion Models) parses the prompt into modifier–noun pairs. During the first 25 denoising steps it pulls each modifier's map onto its noun's map and pushes it away from unrelated words. StructureDiffusion (Feng et al.) likewise works on shallow noun phrases. Both are one-hop methods. Neither states that "zipper" must sit inside "jacket," which must sit inside "man."

Our claim is empirical and falsifiable. Suppose a flat method binds each link correctly with probability $p$, and the links fail roughly independently. Then an attribute at depth $d$ lands on the correct node with probability about $p^d$. Accuracy should decay geometrically with depth. A method that enforces containment along the whole path should decay much more slowly. We also claim that a fixed depth is the wrong knob. Beyond some depth, a part covers too few cells of the attention map, which in Stable Diffusion's middle layers is only 16×16 or 32×32, to be steered separately. Forcing constraints there adds noise rather than control. Generative Scene Graph Networks hard-codes a three-level scene→object→part tree and its authors call for depth set by the input. We give that call a concrete, resolution-based answer in the generative setting.

Research Program

Tree construction. Parser errors on nested syntax are a known failure point for SynGen. We therefore build the tree in two ways and compare them: dependency parsing with rules for possessives, "whose," "with," "on," and "wearing," and a small instructed language model that outputs a part-whole tree over the prompt's tokens. This borrows the root-to-leaf decomposition that hierarchical query decomposition uses in text QA (Patel et al.). There, trees beat linear chains because errors stay inside one branch (Cao et al.). We reuse the idea of isolating branches but apply it to image regions instead of retrieval. AtTGen also binds attributes by generating a tree, but its tree is fixed at two tiers and it only extracts text. It shows that tree-structured binding can be cheap and accurate, not that it transfers to generation.

Guidance objective. Let $A_n$ be the normalized cross-attention map of tree node $n$ at a denoising step. Let $P(n)$ be its parent, $S(n)$ its siblings, and $\bar A_{P(n)}$ the parent map rescaled to peak at one. We minimize

$$ \mathcal{L} = \sum_{n} \Big[ \sum_x A_n(x)\big(1-\bar A_{P(n)}(x)\big) \;+\; \lambda \sum_{s\in S(n)} \sum_x A_n(x)\,A_s(x) \Big], $$

In words, the first term is the attention mass a node places outside its parent. The second term is the overlap between siblings, weighted by $\lambda$. Leaf modifiers such as "gold" are children of their noun. Containment is asymmetric, unlike SynGen's symmetric pull, because a part should lie inside its whole without covering all of it. The objective uses the same number of backward passes per step as SynGen, so its cost should stay close to SynGen's reported 9.7 s against 4.0 s per image. We will measure this rather than assume it.

Depth rule. At each guided step, a child branch stays active only while its parent's map covers at least $k$ attention cells above half its peak. When the parent is smaller than that, its subtree collapses: descendants' attributes attach to the parent as plain modifiers. Depth therefore varies per branch and per image, and a large foreground person can carry four levels while a background person carries one. We will sweep $k$. It is the only new hyperparameter.

Benchmark. Existing suites cannot measure depth. T2I-CompBench (T2I-CompBench; T2I-CompBench++) and the Attend-and-Excite set are mostly one-hop. DetailMaster (DetailMaster) and T2I-CoReBench (Easier Painting Than Thinking) show collapse as scene density grows but do not separate density from nesting.

We therefore build a set of controlled depth-1 to depth-4 prompts, following MuSiQue's bottom-up composition (MuSiQue). We first check that each single-hop binding is renderable on its own, then chain the checked bindings through shared "bridge" parts. Entity count and token length are held fixed across depths, so depth is the only thing that changes. Each prompt has a contrast twin that swaps an attribute between levels, for example a gold zipper with a green pull-tab.

Scoring separates two questions: whether the attribute appears anywhere in the image, and whether it appears on the correct node. These are asked as VQA checklist questions on crops of the detected parent part. The gap between the two answers is our leakage measure. It is the image-generation analogue of MuSiQue's disconnected-reasoning score, and it catches models that paint "gold" somewhere without binding it. A human-rated subset will calibrate the automatic judge.

Baselines

The naive reference is the unguided backbone with the raw prompt. For Stable Diffusion v2, T2I-CompBench reports a B-VQA attribute-binding score of 0.5065. For SDXL, RPG reports a T2I-CompBench color-binding score of 0.6369. We run SD v1.5/2.1, where SynGen's code applies, and SDXL, with identical seeds and step counts.

The strongest method on the one-hop version of this question is SynGen. It is the direct ancestor of our loss, and on the Attend-and-Excite set it reports 94.76% Proper Binding against 81.90% for the prior best method. We run it on the same backbone, seeds, and 25 guided steps. Its learning rate and step size get the same tuning budget as ours, chosen on a held-out slice of our depth-2 prompts. Beating it means matching it at depth 1 (within noise of its A&E score when we reproduce it) and showing a flatter depth curve with a statistically clear margin at depths 3–4 in correct-node accuracy.

The strongest region-isolation competitor is RPG (Mastering Text-to-Image Diffusion), which reports 0.8335 color binding on T2I-CompBench. We run it in its released grid form and in a nested-region variant we build, in which sub-prompts for parts are placed in boxes inside the parent box. That gives it a fair shot at depth. The planner model is the same one that builds our trees.

Among fine-tuning approaches, GORS is the representative, reported at 0.6603 B-VQA on SD v2 in T2I-CompBench. We train it on our backbone with a compute budget matched to our total guidance cost over the evaluation set.

For T2I-CoReBench there are no per-model scores in the file: no reported result found on Lacuna for individual model scores beyond a lag of up to 32 points between open and closed models on reasoning. We use its attribute and dense-composition dimensions only as out-of-distribution checks.

Three ablations isolate the key ingredients. The first is our tree with SynGen's symmetric pull substituted for containment. The second is fixed depth 1, 2, and 3 in place of the adaptive rule. The third is parser trees against language-model trees.

Why not just…?

Why not just run SynGen on every adjacent pair in the parse? Containment might chain for free: pull "gold" onto "pull-tab" and "pull-tab" onto "zipper." We treat this as the most dangerous alternative. It is exactly our symmetric-pull ablation. If it matches the full method across depths, the contribution shrinks to the benchmark and the depth rule, and we will report that. Our expectation that it falls short rests on symmetry. Pulling a small part's map toward a large parent's map also enlarges the part or shrinks the parent, and nothing stops a pull-tab from drifting onto the sleeve, which the parent-bounded term forbids.

Why not just have a language model rewrite the prompt more explicitly? T2I-CoReBench reports that rewriting helps weaker models but gives only marginal gains for strong ones. Ge et al. and Zhuang et al. find that longer prompts make CLIP-style encoders lose adjective-noun links. Rewriting makes nested prompts longer, so it should raise the "anywhere" score but not the correct-node score. We include rewriting as an extra baseline to test that prediction.

Why not just use nested boxes, as a recursive RPG would? This is a real threat for large, well-separated parts. Our nested-region baseline tests it directly. We expect boxes to fail where parts are organic or overlapping, such as a patch on a curved sleeve, and to produce collage seams. We split results by part-overlap ratio to show where each approach wins.

Why not just use a bigger model with a stronger text encoder? IRCoT found that a 3B model with structured retrieval beat a 175B model without it. That is suggestive but comes from QA, not images. We run the unguided depth curve on the largest open backbone we can afford. If its curve is already flat, the problem is solved by scale, and we will say so.

Proposed Experiments

The decisive experiment is the depth curve. For each method we plot correct-node accuracy against depth 1–4 on the controlled benchmark and fit the geometric decay $p^d$. Three results would confirm the claim together. First, the flat methods (SynGen, unguided, rewriting) should fit the geometric decay well. Second, the tree method's per-hop decay should be much smaller. Third, the gap between "anywhere" and "correct-node" should shrink most at depth 3–4.

A second check tests the depth rule on its own. We place identical nested descriptions on large foreground and small background entities. The adaptive rule should beat every fixed depth. Fixed depth 3 should hurt background entities relative to collapsing them, which is evidence that constraints below the attention resolution add noise rather than control.

Third, we confirm no regression at depth 1 on Attend-and-Excite, DVMP, and T2I-CompBench color, shape, and texture. We also report latency against SynGen's measured overhead, and the rate of failures caused by bad trees, comparing parser and language-model trees.

If the depth curves of flat SynGen and the tree method coincide, the central hypothesis is false.