Layout-to-image generators paint a scene from a map of labeled regions ("a red car here, a wet road there"). They share a common failure called attribute leakage: the car's red spreads into the road because every image patch can attend to every other patch. The fix that currently works best is to block attention between patches of different regions in the middle layers of the network and to leave the early and late layers unrestricted, so that global effects such as lighting and shadows can still form. That fix has two costs. Inside each region, attention is still all-to-all, so a large region like sky or road costs quadratic compute. And the boundary is all-or-nothing: a region either sees its neighbors fully or not at all.
We propose to replace the middle-layer masked attention with deformable attention, meaning each patch reads from only a handful of learned, continuously placed sample points rather than from every patch. The sample points are confined to the patch's own region by construction. A small, explicitly budgeted set of "harmony points" is allowed to cross the boundary. The offsets are produced by a per-region generated kernel, so a thin lamp post and a sprawling sky learn different sampling patterns. The claim fits in two sentences. Leakage and harmony can be traded against each other through a per-region sampling budget, a finer control than choosing which layers to mask. The same change turns the within-region cost from quadratic in region size into linear in the number of patches.
Core Claim and What Is New
The closest work is Seg2Any. It adds an Attribute Isolation Attention mask to the FLUX diffusion transformer, a large image generator built from transformer blocks. The mask blocks self-attention across entity boundaries only in layers 20 to 38. The authors report that masking all layers destroys shadows and reflections. We take that finding as a constraint, not something to overturn. Seg2Any keeps dense attention inside each entity and treats the boundary as a hard wall. We replace both properties.
The second anchor is MADIN. For image inpainting, it steers deformable sampling offsets away from corrupted pixels using a binary mask. It uses masks only to exclude bad regions and never to bind an entity to its own region. It also has no notion of several entities with competing attributes. We reuse its insight that offsets must be mask-aware. We change the purpose from avoidance to confinement plus a controlled cross-boundary budget.
The third ingredient comes from dynamic kernel generation. CondInst synthesizes a 169-parameter mask head per instance from a detection embedding plus relative coordinates. We borrow that recipe to synthesize, per entity, the small projection that predicts offsets and weights. The alternative is sharing one static projection across all entities, as Deformable DETR does.
We know of no paper in our search that combines mask-confined deformable sampling, per-entity generated offset kernels, and a cross-boundary budget inside a generative transformer.
Mechanism
Consider a middle layer and a latent image patch $q$ belonging to entity $m$, with reference location $p_q$. An entity-specific controller takes the entity's caption embedding, the entity's mask geometry, and relative coordinates. From these it generates the weights of a tiny linear map. That map sends the patch feature to $K_{in}$ inside offsets and $K_{out}$ harmony offsets, each with an attention weight.
Inside points are snapped into the entity mask. We precompute a signed distance field of the mask and move any point that falls outside back along the distance gradient. As a result, inside samples cannot read a neighbor's features, however the offsets are predicted. Harmony points are unconstrained, but their weights pass through a separate softmax whose total mass is capped by a learned per-entity scalar. During training, an L1 penalty on that scalar pushes cross-boundary reading toward the minimum needed.
The output mixes bilinearly sampled values in the usual deformable way. Per layer, the within-region cost drops from $\sum_m N_m^2$ to $N(K_{in}+K_{out})$. Here $N_m$ is the number of patches in entity $m$, $N$ is the total patch count, and $K_{in}+K_{out}$ is typically 8. Early and late layers remain dense and global, exactly as in Seg2Any.
Research Program
The work has three parts, run in order. First, we build the module as a drop-in replacement for the Attribute Isolation Attention layers. We fine-tune on SACap-1M with Seg2Any's recipe, keeping its shape adapter and caption alignment masks unchanged so the only varied component is the middle-layer attention.
Second, we map the leakage–harmony frontier by sweeping $K_{out}$ from 0 to 3 and the budget cap. The central scientific question is whether $K_{out}=0$ reproduces the "pasted-on" composites that hard masking produces, and whether one or two harmony points restore contact shadows without reintroducing leakage.
Third, we test whether the budgeted boundary lets masking extend beyond Seg2Any's layer 20–38 sweet spot. If harmony points carry the lighting information that dense early and late layers currently carry, masking could be widened. That would matter for compute, because an efficiency gain limited to a third of the blocks is capped by Amdahl's law.
We will measure efficiency in wall-clock time, not FLOPs. Deformable DETR itself notes that irregular bilinear lookups lag their theoretical savings on GPUs. We expect wall-clock gains only at high resolution, meaning 2048-pixel images and above, where a single large entity makes $N_m^2$ dominant. At 1024 pixels, FLUX has only about four thousand latent patches, so dense attention is cheap there. We state this as a limitation, not a hidden assumption.
Baselines
The naive reference point is the same FLUX backbone fine-tuned with Seg2Any's shape conditioning but with no attention mask in any layer. This shows how much leakage there is to remove.
The strongest existing method is Seg2Any with its middle-layer isolation mask. It reports 94.90 class-agnostic MIoU on SACap-Eval and 45.54 MIoU on COCO-Stuff. We will retrain it ourselves on identical data, steps, resolution, and hyperparameter search budget, rather than rely only on the reported figure. Beating it means matching its MIoU within noise across three seeds while showing lower leakage and equal or better harmony scores. If we cannot reduce leakage, then only a high-resolution wall-clock win would count.
A third baseline is Seg2Any's exact masking executed as block-sparse attention. Patches are reordered by entity so the mask becomes block-diagonal, and cross-entity blocks are skipped. This follows the block-skipping idea that Trainable Dynamic Mask Sparse Attention implements for language. Its output is identical to Seg2Any's, so it is the honest speed baseline.
Four ablations isolate our ingredients. The first uses unconstrained deformable attention in the middle layers, the Deformable DETR design, to test whether confinement matters. The second is confined sampling with $K_{out}=0$, to test whether harmony points matter. The third replaces per-entity generated offset kernels with one static shared projection, to test whether the dynamic kernel matters. The fourth is a within-mask local-window variant, to test whether deformation matters or simple locality suffices.
For older reference points on COCO-Stuff, we cite FreestyleNet, which applies hard out-of-mask logit masking and reports 14.4 FID. We also cite the training-free DenseDiffusion, which modulates attention early in denoising and reports 34.99 IoU on its 250-sample dense caption split. These are reported numbers, not retrained comparisons.
Why not just…?
Why not just run Seg2Any's mask as block-sparse attention? This is the most serious threat. It gives exact Seg2Any quality at a cost of $\sum_m N_m^2$, which beats dense attention whenever entities are numerous and small. Our method can only win on speed when one or a few entities dominate the image at high resolution. It can only win on quality if the budgeted boundary improves harmony or leakage. We will report the crossover resolution and the largest-entity fraction at which each method is faster. If block-sparse attention is faster everywhere we test and our quality is merely equal, the efficiency half of the claim fails. In that case the contribution reduces to the leakage–harmony control, and we will say so.
Why not just keep Seg2Any's layer schedule and accept dense attention? At 1024 pixels this may well be adequate, and our wall-clock results may show that. The schedule is still a coarse control: every masked layer is fully walled and every unmasked layer is fully open. Seg2Any provides no way to let a car's shadow reach the road in a masked layer while its color stays out. The $K_{out}$ sweep directly tests whether a finer control exists and whether it lets masking extend to more layers.
Why not just use unconstrained deformable attention or hierarchical sparse attention in those layers? Unconstrained offsets freely cross boundaries. MADIN shows that mask-agnostic deformable sampling pulls in the wrong features, and our first ablation measures the resulting leakage directly. Trainable Log-linear Sparse Attention reaches 20.41 FID on ImageNet-256 with $O(N \log N)$ hierarchical selection, but it pools over rigid grids that ignore entity identity. We would expect leakage similar to the unmasked baseline unless its pooling is restricted to entity masks. Doing that would be a reasonable additional baseline if time permits.
Proposed Experiments
The main evaluation uses SACap-Eval class-agnostic MIoU and COCO-Stuff MIoU and FID.
For leakage we add a new measurement. For each entity, we crop the generated region and ask a vision-language judge to pick among that entity's caption and its neighbors' captions. The leakage rate is the fraction of crops matched to a neighbor. No reported result exists on Lacuna for this metric, so all methods are measured fresh.
For harmony, we build a curated subset of SACap-Eval layouts with contact relations, such as objects on surfaces or people in sunlight. Blinded human raters give pairwise judgments of "pasted-on" appearance, and a vision-language judge answers the same question for scale.
For efficiency, we time the middle-layer attention and full sampling runs at 1024 and 2048 pixels on the same GPU. We report results against both dense Seg2Any and the block-sparse variant, binned by the largest entity's area fraction.
Three decisive checks follow from the design. The first is whether $K_{out}=0$ reproduces hard-mask artifacts while $K_{out}\in\{1,2\}$ removes them at unchanged leakage. The second is whether per-entity generated kernels beat the static projection most on thin or irregular entities, where a shared projection cannot see mask shape. The third is whether confined sampling with harmony points permits masking beyond layers 20 to 38 without harmony loss. A negative answer on the first check would falsify the core claim.
References
- Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control
- Learning Mask-Aware Offsets: Two-branch Deformable Attention Networks for Inpainting with Masked Region Avoidance
- Instance and Panoptic Segmentation Using Conditional Convolutions
- Deformable DETR: Deformable Transformers for End-to-End Object Detection
- Trainable Dynamic Mask Sparse Attention
- Freestyle Layout-to-Image Synthesis
- Dense Text-to-Image Generation with Attention Modulation
- Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers