Closed-Loop Attention Boosting for Layout-Controlled Diffusion

Text-to-image diffusion models generate an image by gradually removing noise from a random starting point, guided by a text prompt. When the prompt names several objects with attributes, as in "a blue apple and a green vase," they often bleed one object's color onto another or drop an object entirely. A popular training-free fix takes a user-drawn layout mask for each object and boosts each word's attention inside its region while suppressing it outside. These methods work, but they depend on a hand-set boost strength and a hand-picked window of denoising steps. Their authors report that both must be retuned for each prompt and layout. In practice, the knob that makes layout control work is the same knob that makes it brittle.

We propose treating the boost as a feedback controller instead of a constant. At every denoising step we measure a scale-free quantity: the fraction of each word's cross-attention mass that falls inside its assigned mask. A proportional-integral (PI) controller, the kind of rule a thermostat uses, then raises or lowers that word's boost to drive this fraction toward a target. The user no longer picks a boost strength whose meaning depends on the backbone, the token, and the mask size. They pick a target containment level, such as "90% of 'apple' attention inside the apple mask." The claim we test is concrete. A single target setting should transfer across prompts, mask sizes, and backbones, where the best fixed boost does not, and it should do so without losing image quality.

Core Claim

The open problem is not how to mask attention. Hard masking, boosting, self-attention suppression, and potential fields all exist. The open problem is that every inference-time method carries a strength parameter whose best value moves with the sample. Localized Text-to-Image Generation for Free via Cross Attention Control (CAC) masks and amplifies cross-attention with a boost $\lambda$ and a fixed timestep window, and notes that both need manual calibration. This paper motivates our work. Dense Text-to-Image Generation with Attention Modulation (DenseDiffusion) recognizes the same problem and answers with fixed formulas. It gives small masks a larger boost and keeps boosted values within the original value range, over a hardcoded window $t\in[0.7,1.0]$. This is our closest competitor: it adapts, but open-loop. It cannot react when a particular small object fails to appear. AnyCanvas replaces binary masks with a continuous signed-distance potential that pushes attention centroids into place. It is also open-loop, and it breaks on fragmented masks. Our proposal differs from all three because the boost at step $t$ depends on the measured outcome at earlier steps rather than on a precomputed schedule.

The method comes from multi-view clustering. Deep matrix factorization with adaptive weights for multi-view clustering (DMFAW) replaces a grid-searched feature weight with PI step-size control driven by recent loss changes. It reports convergence in under 10 iterations and a 13x runtime saving over the fixed-parameter version on ORL. We reuse the idea of a PI controller taking over from a grid-searched trade-off weight, not that paper's objective. The choice of controlled signal is constrained by Beyond Magnitude: Scale-Invariant Evidential Fusion for Multi-View Classification. That paper shows raw logit magnitudes let loud inputs dominate fusion regardless of real certainty. For us, this means the controller must act on a normalized quantity (attention mass, which lies in $[0,1]$ by construction) rather than on raw logits. Raw logits differ in scale across tokens with different embedding norms and across backbones.

Research Program

At denoising step $t$, let $A_{t}$ be the cross-attention map of text token $j$ averaged over heads and layers at the chosen resolution. Let $M_j$ be its binary layout mask. The measured containment is

$$m_{j,t} = \frac{\sum_{i \in M_j} A_{t}(i,j)}{\sum_{i} A_{t}(i,j)},$$

which is the share of token $j$'s attention landing inside its region. The controller adds a log-boost $b_{j,t}$ to the in-mask logits of token $j$ and updates it as

$$b_{j,t-1} = b_{j,t} + k_p\, e_{j,t} + k_i \sum_{\tau \ge t} e_{j,\tau}, \qquad e_{j,t} = s^\ast - m_{j,t},$$

where $s^\ast$ is the user's target containment and $k_p, k_i$ are controller gains. The error $e$ is bounded and unitless, so the gains should not need rescaling across models. Whether that holds is an empirical question we test rather than assume. Control turns off once $m_{j,t}$ stays near $s^\ast$, or at a step-budget cutoff. This replaces the hand-picked timestep window with a stopping rule. Global style tokens are left uncontrolled, as in Freestyle Layout-to-Image Synthesis, which exempts style tokens from its out-of-mask $-\infty$ rectification.

We build this as a drop-in replacement for the boost in CAC and in DenseDiffusion's cross-attention path, on Stable Diffusion first. We then port it unchanged to SDXL and to a transformer backbone. The transformer port is where Seg2Any found that self-attention isolation helps only in middle layers, so we control cross-attention only and leave self-attention as in each baseline.

The central analysis is a sensitivity study, not just a leaderboard. For every method we sweep its strength parameter ($\lambda$ for CAC, $s^\ast$ and gains for ours) and report three things. The first is the best score. The second is the width of the parameter range within a fixed tolerance of the best. The third is how far the per-stratum optimum moves across strata defined by mask area, number of objects, and backbone.

Baselines

The do-nothing reference is the unmodified base model given the same prompt. It shows how much any layout control helps; CAC reports CC-500 attribute-binding success rising from 49.91% for the base model to 67.27% with CAC.

The strongest fixed-parameter competitor is CAC itself (Localized Text-to-Image Generation for Free via Cross Attention Control). We give it a fair tuning budget: its $\lambda$ and timestep window are grid-searched on a held-out development split with the same number of generated images we spend tuning $s^\ast$, $k_p$ and $k_i$. Its reported numbers are the 67.27% CC-500 binding success above and an object-generation precision on COCO of 0.257, up from 0.094.

The strongest adaptive competitor is DenseDiffusion (Dense Text-to-Image Generation with Attention Modulation). Its mask-area and value-range rules are exactly the open-loop alternative to feedback. It reports an IoU of 34.99 on its 250-sample dense-caption COCO subset, against 23.76 for SD-Pww. We run it with its published settings and also with its global strength retuned on our development split.

We include an inference-time hard-mask variant in the spirit of FreestyleNet's $-\infty$ rectification. It has no strength knob, so it marks the "maximum containment, ignore quality" end of the trade-off. On the Zero-Painter 3,000-layout COCO set we report Zero-Painter's own figures (IoU 0.75, local CLIP 26.68, from Zero-Painter) as a multi-pass reference, since that method spends several passes per image.

The key ablation separates feedback from scheduling. We record the average boost trajectory our controller produces on the development set, then replay that fixed schedule open-loop on test samples. A second ablation removes the integral term. A third swaps the normalized error for raw-logit error, testing whether scale-freeness is what buys transfer.

Beating the baselines means the following. Within the same tuning budget, the controller should match or exceed tuned CAC and DenseDiffusion on IoU and binding accuracy, with FID and CLIP score no worse. It should also show a clearly wider near-optimal parameter range and a smaller shift in optimal setting across strata. Matching peak score while greatly reducing sensitivity counts as success, because sensitivity is the claim.

Why not just…?

Why not just grid-search $\lambda$ once per backbone and ship it? This is the real threat. If per-sample optimal $\lambda$ barely varies, a single tuned value captures nearly everything and feedback adds nothing. We settle it first, before building anything else. On the development split we compute an oracle per-sample best $\lambda$ for CAC and compare it with the best global $\lambda$. If the oracle gap in IoU and binding accuracy is small, we abandon the controller and report that negative result. The multi-view evidence that tuned weights fail to transfer (Deep matrix factorization with adaptive weights for multi-view clustering) only motivates the test; it does not prove it holds for diffusion.

Why not just make $\lambda$ a function of mask area, as DenseDiffusion does? Area is the most obvious predictor of the right boost, and a regression of oracle $\lambda$ on area and object count is cheap. We fit exactly this regression on the development split and treat it as a baseline. The controller is worth having only if a meaningful share of the oracle gap remains unexplained by area. That remainder would come from things a static rule cannot see, such as a word whose attention is unusually diffuse in a particular prompt. We report how much of the oracle gap the area rule closes, versus the controller.

Why not just hard-mask with $-\infty$ and drop the knob entirely? Hard masking has no knob but is not free. FreestyleNet documents artifacts on small masks and failure on counterfactual layouts. Seg2Any shows that blanket hard isolation destroys global lighting and reflections. Our hard-mask baseline will show whether it simply wins on containment and loses on FID. A target $s^\ast<1$ is the obvious way to buy back realism.

Doesn't the controller just move the tuning into $k_p$, $k_i$ and $s^\ast$? Yes, it adds three numbers, so the claim has to be that they are less sensitive than $\lambda$, not that they vanish. Because the error is bounded in $[-1,1]$, the gains are dimensionless. We test directly whether gains tuned on Stable Diffusion transfer to SDXL and the transformer backbone without retuning, and compare against $\lambda$ transferred the same way.

Proposed Experiments

We evaluate on CC-500 for attribute binding, on DenseDiffusion's 250-sample dense-caption COCO subset for IoU, SOA-I and CLIP score, on COCO 2017 layouts for CAC's object-precision metric, and on Zero-Painter's 3,000 COCO layouts for IoU and local CLIP. Image quality is measured with FID against COCO images. We stratify every benchmark by mask area and by object count. Results on the smallest-mask stratum are reported separately, since both DenseDiffusion and FreestyleNet identify small objects as the failure point.

The attention containment signal can be gamed: attention can sit inside the mask while the pixels do not. So all spatial scores are computed on the output image with an off-the-shelf detector or segmenter, never on the attention maps the controller steers. We also report how well final $m_{j,T}$ correlates with output IoU. A weak correlation would mean we are controlling the wrong quantity.

The experiments run in a fixed order, and three checks decide the outcome. First is the oracle-gap test from the grid-search paragraph above; it determines whether the project continues. Second is the replayed-schedule ablation, which tells us whether any gain comes from feedback or merely from a better fixed schedule. Third is the zero-retuning transfer to SDXL and the transformer backbone, which tests the sensitivity claim where it matters most to practitioners. If the controller wins only on the first backbone and needs retuning on the others, it has not solved the hyperparameter problem, and we will say so.