Token-Sparse Implicit Steps for Layout-Controlled Diffusion Sampling

Text-to-image diffusion models generate a picture by integrating a differential equation that slowly turns noise into an image. To make them obey a layout ("a blue apple here, a green vase there"), the most popular training-free methods edit the model's attention. Attention is the operation by which each image patch decides which words and which other patches to draw information from. These methods block it across region boundaries, a technique called spatial attention masking. The masking works, but it makes the equation harder to integrate. The masks switch on and off during sampling and impose sharp boundaries. As a result, the vector field changes abruptly in a few places while staying smooth almost everywhere else. Numerical analysts have a name for this situation: localized stiffness. Their standard remedy is an implicit step, which solves an equation for the next state rather than extrapolating to it. Implicit steps are expensive, and in a diffusion model every solver iteration costs a full network evaluation.

This note proposes making the implicit step token-sparse. Each sampling step solves the implicit update by fixed-point iteration. After every iteration we measure, separately for each image token, whether its update has converged. Only the unconverged tokens are re-evaluated in the next iteration. In a diffusion transformer this is itself an attention mask: unconverged tokens issue fresh queries, while converged tokens contribute cached keys and values. The bet is that solver error under layout masking concentrates in a small fraction of tokens, mostly near region boundaries and inside small objects. If so, most of the layout-adherence gain of a full implicit corrector can be bought for a fraction of its cost, at the low step counts people actually use.

Core Claim

At 8–20 network evaluations, masked layout samplers lose much of their attribute-binding and boundary accuracy. A large part of this loss comes from integration error that is spatially concentrated. A per-token convergence test inside an implicit predictor-corrector step can find and fix that error while re-evaluating only a minority of tokens. What would be new is using spatial attention masking to set the computational support of the implicit solve, rather than only the semantic support of each prompt token.

Write one sampling step of size $h$ as the implicit equation $x = x_t + h\,\Phi(x)$. Here $x_t$ is the current latent and $\Phi$ is the solver's update built from the network's noise prediction. We solve it by the iteration $x^{(k+1)} = x_t + h\,\Phi(x^{(k)})$, initialized from an explicit multistep predictor. The per-token residual is $r_i^{(k)} = \lVert x_i^{(k+1)} - x_i^{(k)} \rVert$ for token $i$. The active set for the next iteration is

$$A^{(k+1)} = \{\, i : r_i^{(k)} > \tau \,\},$$

where $\tau$ is a tolerance. Only tokens in $A^{(k+1)}$ recompute their queries and feed-forward layers. All other tokens are frozen and expose their cached keys and values. The iteration stops when $A$ is empty or after a cap of $K$ iterations. Two costs scale roughly with the active fraction $|A|/n$: query and feed-forward computation scale with it directly, and attention scales with it times $n$. This is why the payoff depends on the concentration hypothesis.

Relation to Prior Work

The layout-control side supplies the setting and the failure. Cross Attention Control, in Localized Text-to-Image Generation for Free via Cross Attention Control, restricts each region's pixels to its own localized prompt at under 0.1 s of overhead. Dense Text-to-Image Generation with Attention Modulation (DenseDiffusion) also blocks self-attention between segments, but only in the first 30% of denoising, $t \in [1.0, 0.7]$. That time-switched schedule is exactly the kind of abrupt change in the vector field that hurts multistep solvers. Seg2Any restricts self-attention only in the middle layers of a diffusion transformer, to keep global lighting coherent. Zero-Painter and Freestyle Layout-to-Image Synthesis document seam artifacts and hard-mask behavior at boundaries. All of these treat the sampler as a fixed black box, typically DDIM or explicit multistep. We keep their masks unchanged and change only how the sampler integrates under them.

AnyCanvas replaces hard masks with smooth signed-distance potentials. It reports instability on fragmented masks such as checkerboards and thin lattices. This constrains our design in two ways. Smoothing is a credible competing fix that we must test against. Fragmented masks are a regime where we expect localized correction to matter most.

The solver side supplies the method. ERA-Solver uses an implicit Adams predictor-corrector with an error-robust selection of history terms driven by the measured noise-prediction error. That error is computed globally over the whole latent and was evaluated without any layout control. We reuse its predictor-corrector structure but compute the error per token and use it to choose where to iterate, not which history to keep. DPM-Solver and DPM-Solver++ provide the explicit high-order predictors and the strongest explicit baselines.

The numerical-PDE papers motivate the locality argument rather than supplying code. Efficient Simulation of Singularly Perturbed Systems Using a Stabilized Multirate Explicit Scheme shows that resolving fast transients briefly before a large step collapses cost, from 5,000,000 to 1,780 steps. Its sub-steps apply to the whole state vector. The primal-dual solvers in A primal-dual approach for solving conservation laws with implicit in time approximations and A first-order computational algorithm for reaction-diffusion type equations via primal-dual hybrid gradient method show that implicit steps can be solved with pointwise, GPU-parallel first-order updates and no Jacobian. That is the property that makes per-token freezing well defined in our fixed-point iteration.

Research Program

The first stage is diagnostic and decides whether the project continues. We take an open diffusion transformer of the kind Seg2Any builds on, apply CAC-style cross-attention masking and DenseDiffusion-style self-attention blocking, and sample with DPM-Solver++ at 10 evaluations. For each step we compute a reference next state using many small steps. We then measure how the explicit step's error is distributed across tokens. Specifically, we record what fraction of total error mass falls in the top 10% of tokens, and how much of it lies within a fixed-width band around mask boundaries. We repeat the measurement with masking off. If masking does not make error both larger and more concentrated, the core claim fails and we stop.

The second stage builds the token-sparse corrector. The predictor is a DPM-Solver++ multistep step. The corrector is an Adams-Moulton-type implicit step in the style of ERA-Solver. Each corrector iteration is a forward pass in which inactive tokens skip their queries and feed-forward layers and their cached keys and values are reused. We study the tolerance $\tau$ (or, equivalently, a fixed active fraction $\rho$) and the iteration cap $K$. We also study when cached keys and values go stale, by testing one refresh per step against no refresh. Cost is reported in measured FLOPs and wall-clock time, not only in network evaluations. A partial evaluation is not a full evaluation, and counting it as one would hide the result.

A secondary stage checks U-Net backbones such as the Stable Diffusion models used by CAC and DenseDiffusion. Convolutions make true token sparsity awkward there. On those models we expect only quality results, not savings, and we report them as such.

Baselines

The naive reference is the masked layout method run with its default explicit sampler (DPM-Solver++ multistep) at the same number of steps, with no corrector. Its layout scores at 10–20 steps establish how much accuracy masking loses at low step counts. For orientation, the original papers report CAC at 67.27% compositional success on CC-500 against 49.91% for the base model. DenseDiffusion reports IoU 34.99 against 23.76 for SD-Pww on 250 MS-COCO dense descriptions. Zero-Painter reports IoU 0.75 and local CLIP 26.68 on 3,000 MSCOCO layouts. For these methods at reduced step counts, or combined with an implicit corrector, no reported result was found on Lacuna. We rerun every method ourselves under identical prompts, layouts, seeds, and backbone.

The strongest existing competitor is a global implicit corrector: ERA-Solver-style predictor-corrector applied to the entire latent, with the same masks. It directly answers whether locality adds anything beyond implicitness. ERA-Solver reports FID 3.54 at 10 evaluations on CIFAR-10 in its results section (5.14 in its abstract) and 9.42 on LSUN-Church. These are unconditional settings, so they only confirm that the solver works, not that it helps layouts. We run it with the same iteration cap $K$ and tune its hyperparameters with the same budget as ours. Beating it means two things: matching its layout scores within seed noise at no more than half its FLOPs, or exceeding it at equal FLOPs.

A compute-matched explicit baseline spends our method's measured FLOPs on extra DPM-Solver++ steps instead. If spending the budget on more steps does as well, token-sparse correction has no reason to exist.

Two ablations isolate the key ingredient, which is residual-driven token selection. The first selects a random token subset of the same size. The second uses a static boundary band computed from the layout alone before sampling, which is free to obtain. Our method must beat random selection clearly. To justify the adaptive residual test, it should also beat the static band, at least on fragmented and overlapping layouts.

Finally, we test AnyCanvas-style smooth masks and a smoothly ramped version of the DenseDiffusion schedule, each with the plain explicit sampler. If removing the stiffness at its source works as well as correcting it, we want to know that.

Why not just…?

Why not just take more sampling steps? Extra steps spend compute uniformly across every token and every time point. Our diagnostic stage predicts that the error lives in a small fraction of tokens. The compute-matched explicit baseline settles this directly. If extra steps match token-sparse correction at equal FLOPs, the concentration hypothesis is practically irrelevant and we would report that.

Why not just run a global implicit corrector like ERA-Solver? It probably recovers comparable quality, so this is a cost question, not a quality question. A global corrector iteration costs a full network evaluation, and that cost is exactly the implicit-solver expense the PDE literature works to avoid. The comparison above measures how much of the global corrector's gain survives when only, say, 10–30% of tokens are re-evaluated. If the active fraction needed for parity turns out to be near 100%, the idea fails.

Why not just smooth the masks or the on/off schedule so there is no stiffness to correct? This is the most serious threat. If the low-step failure comes mainly from the abrupt switch-off at $t = 0.7$, a smooth ramp might fix it for free. AnyCanvas's own reported instability on fragmented masks suggests smoothing does not remove localized difficulty everywhere. Still, that evidence comes from potential-field guidance, not from schedule ramps. We therefore run smoothed schedules as baselines and also combine them with our corrector. The honest prediction is that smoothing captures part of the gain on simple box layouts and that the corrector's advantage concentrates on fragmented, overlapping, and small-object layouts. If smoothing alone matches the corrector even there, the contribution reduces to a schedule fix and we would say so.

Why not just pick the boundary band from the layout instead of measuring residuals? If error sits only at mask edges, a static band is simpler and costs nothing to compute. The static-band ablation tests this. A tie on box layouts would still be a useful negative-on-novelty result, since it would tell practitioners where to spend correction. Adaptivity is justified only if residual selection wins where error appears away from edges, as in small objects being suppressed or global lighting conflicts.

Why not just fine-tune a layout adapter? Training-based control is a different trade-off. It requires paired data, and Zero-Painter and Freestyle Layout-to-Image Synthesis argue it generalizes worse to open-set prompts. It also does not remove the integration question, because a fine-tuned model is still sampled at low step counts. We leave it out of scope rather than claim superiority.

Proposed Experiments

The benchmarks cover attribute binding and spatial precision. Attribute binding is scored on CC-500 with the detector-based success rate used by CAC, and on the attribute-binding split of T2I-CompBench with its B-VQA score; that paper reports 0.5065 to 0.6603 for GORS fine-tuning on Stable Diffusion v2, a training-based reference point. Spatial precision is scored with IoU on DenseDiffusion's 250 MS-COCO dense descriptions and with IoU and local CLIP on Zero-Painter's MSCOCO layouts. We add a stress set of fragmented masks (checkerboards, thin lattices) and overlapping masks, built from the failure cases AnyCanvas and Zero-Painter describe, with layout IoU as the metric. Global image quality is tracked with FID on COCO prompts so that gains in adherence cannot hide degradation elsewhere.

We sweep 8, 10, 15, 20, and 50 steps and report every result as a curve of quality against measured FLOPs and wall-clock time, not just evaluation count.

Three checks are decisive. First, the diagnostic must show that masking raises explicit-step error and concentrates it, with a large share of error mass in a small share of tokens. Second, on the quality-against-FLOPs curve, the token-sparse corrector must sit above both the compute-matched explicit baseline and the global corrector at 10–15 steps. Third, residual-driven selection must beat random selection everywhere, and beat the static boundary band on the fragmented and overlapping stress set.

If the first check fails, the premise is wrong. If the second fails, implicit correction is not worth its cost here. If only the third fails, the result narrows to "correct a fixed boundary band," which is still a practical finding.