Protecting the Coarse Layout Inside Diffusion Transformers with Attention-Graph Wavelets

Image generators that build a picture by gradually removing noise (diffusion models) often get every local patch right and the whole picture wrong. They produce dogs with five legs, two heads on one body, horizons that do not line up. Theory for convolutional diffusion models explains the pattern. Early in sampling, the network effectively sees the whole image and decides the coarse layout. Late in sampling, its effective view shrinks until it is a local patch-matcher that can stitch together locally plausible joints into impossible bodies (An analytic theory of creativity in convolutional diffusion models). Graph neural network research has studied the mirror image of this problem for years. When nodes repeatedly average their neighbours' features, the smooth, low-frequency part of the signal takes over and node identity is lost. This is called over-smoothing. That literature has precise tools for separating a signal on a graph into coarse and fine bands.

This note proposes using those tools inside the self-attention layers of a Diffusion Transformer (a diffusion model whose denoiser is a Transformer over image patches, or DiT). The aim is to diagnose and repair late-stage structural failures. The claim is that these failures come from too little protection of the coarse band, not from too much smoothing. The coarse band is the part of the patch features that varies slowly over the attention graph. It is set early in sampling and then allowed to drift late, after the network has stopped looking globally. We split each attention layer's output into multiscale graph wavelet bands, built from the attention map itself. During late reverse steps, we pin only the coarse bands to a cached early-step state, and we do this only at the patches where they are drifting. The method needs no training, and because the wavelets are spatially localized, it can fix one region without touching the rest. That is exactly the limitation the strongest existing frequency-domain method concedes.

Core Claim

Take one DiT block at reverse step $t$ and let $A_t$ be its head-averaged, row-stochastic attention matrix over the $N$ image tokens. Treat $A_t$ as a weighted directed graph, and let $P_t = (I + A_t)/2$ be its lazy random walk. Following the diffusion-wavelet construction used in Geometric Scattering Attention Networks and Molecular Graph Generation via Geometric Scattering, define band operators as follows:

$$\Psi_0 = I - P_t,\qquad \Psi_j = P_t^{2^{j-1}} - P_t^{2^{j}}\ (1\le j\le J),\qquad \Phi_J = P_t^{2^J}.$$

These operators telescope, so $\Psi_0 + \dots + \Psi_J + \Phi_J = I$. They are an exact decomposition of the token features $X$ into a fine band $\Psi_0 X$, intermediate bands, and a coarse band $\Phi_J X$. This generalizes the two-channel split $I = L_{LP} + L_{HP}$ of Complete the Missing Half to many scales. Unlike a global cosine transform, each row of $P_t^{2^j}$ has bounded support on the attention graph.

The intervention operates during late steps, $t < t_0$. We take the bands at scales $j \ge k$ and blend them toward the corresponding bands of features cached at an early step $t_c$, the point where the receptive field is still global:

$$\tilde X = \sum_{j<k}\Psi_j X + \Big(\sum_{j\ge k}\Psi_j + \Phi_J\Big)\big(X + \Lambda_t (X_{t_c} - X)\big).$$

Here $\Lambda_t$ is a diagonal per-token gate. It grows with that token's own coarse-band drift $\|(\Phi_J (X - X_{t_c}))_i\|$ and with how late the step is. Fine and intermediate bands evolve freely, so texture is never frozen. Coarse structure is anchored only where it is moving.

This position treats both sides of the over-smoothing debate seriously. Setting the Record Straight on Transformer Oversmoothing showed that forcing a 48-layer Transformer to smooth helped CIFAR100 accuracy more than forcing it to sharpen (+2.62% versus +1.81%). Position: Don't be Afraid of Over-Smoothing And Over-Squashing argues against reflexively removing smoothing. We take the same view: low-pass content is the carrier of global coherence, and it should be protected rather than suppressed.

How This Differs From the Closest Work

Two papers supply the motivation. Kamb and Ganguli's analytic theory (Kamb et al.; An analytic theory of creativity in convolutional diffusion models) predicts trained ResNet samples with median $r^2$ of 0.94 to 0.96. It predicts self-attention backbones much less well ($r^2 \sim 0.77$), and it leaves attention without a closed-form account. We use that gap as a diagnostic target: if attention breaks the patch-mosaic picture, its departure from the local theory should live in the coarse graph bands.

Beyond Pixel Space supplies the empirical hook. Structural failures show elevated Monte Carlo uncertainty in low-frequency cosine-transform bands, and penalizing that uncertainty improved a 128×128 ImageNet U-Net from FID 15.76 to 13.21. That method acts on pixels and outputs, uses a global basis that its authors say causes non-local side effects, needs extra network evaluations for its Monte Carlo estimates, and names spatially localized wavelets as an open direction. We act inside the network, on a basis defined by the model's own attention graph, with one cached forward state and no sampling.

Setting the Record Straight supplies the operator view of attention, $X_\ell \leftarrow X_{\ell-1} + A X_{\ell-1} H$. Its smoothing and sharpening controls are trained, applied uniformly across inputs, and evaluated only on discriminative models. Ours are training-free, depend on the timestep, are local per token, and are evaluated on generation.

Tragakis et al. also schedule a transition from structural guidance to free texture generation. Their schedule acts on external guidance, whereas ours acts on internal feature bands. Whether the band split matters beyond scheduling is something our ablations test directly.

Research Program

The work has three parts, and each can stand alone.

The first part is a measurement. Existing over-smoothing metrics such as Dirichlet energy average uniformity over all nodes and cannot tell useful clustering from collapse (Are we measuring oversmoothing in graph neural networks correctly?). We replace them with a per-band, per-token drift profile. For each block and each timestep, we record the energy in each wavelet band and the drift of each band relative to the cached state. We compute this on pretrained DiT-XL/2 at ImageNet 256×256 and on the self-attention backbones from the analytic-creativity setting (CIFAR10, CelebA). In the small setting, we also test whether the residual between the attention model's output and the local-score theory's prediction concentrates in the coarse bands.

The second part is the intervention described above. We choose $t_0$, $t_c$, $k$, and $J$ on a held-out class split, and we cap the tuning budget so that it matches the budget given to each baseline.

The third part is localization. We locate failure regions, either from rater annotations or from where coarse-band drift is highest. We then measure how much of each intervention's pixel change falls outside those regions. This directly tests the non-local bleed that Beyond Pixel Space reports.

Baselines

The naive reference is unmodified DiT-XL/2 sampling with its standard sampler and guidance scale. The file reports DiT-XL/2 at FID 2.27 on ImageNet 256×256, via DiCo. We will re-run it with our seeds and sampler so that every comparison uses identical noise. Beating it means a lower structural failure rate, with FID within run-to-run noise of 2.27 or better.

The strongest structure-specific competitor is frequency-domain uncertainty guidance from Beyond Pixel Space. Its reported number is ImageNet 128×128 U-Net FID 15.76 → 13.21. The file has no reported result on Lacuna for this method on DiT at 256×256, so we will run it ourselves on the same DiT checkpoint and seeds. It will get the same hyperparameter-search budget we use and its native number of Monte Carlo samples. We will also report a variant matched to our cost in network evaluations. Beating it means a lower failure rate at equal or better FID, with less change outside the failure region. A fair outcome to state in advance: if it achieves a lower failure rate but larger out-of-region change, the result is a trade-off rather than a win.

Scheduled structural guidance in the spirit of Tragakis et al. is the scheduling competitor. We will implement it as pulling the full late-step prediction toward the early-step prediction on the same $t_0$ schedule.

Two ablations isolate the key ingredient. The first is full-feature anchoring: the same $\Lambda_t$ applied to all bands, with no wavelet split. It tests whether separating bands matters. The second is global-basis anchoring: the same blend, but with bands defined by a 2D cosine transform over the token grid instead of attention-graph wavelets. It tests whether localization on the attention graph matters. A third ablation sets $\Lambda_t$ to a scalar, which tests whether per-token gating matters.

Why not just…?

Why not just raise attention temperature or lower guidance late in sampling? Sharpening or flattening $A_t$ changes how far each token looks, but it does not separate the layout already decided from the texture still being formed. Still, it is the cheapest knob available and a real threat. We include a per-timestep temperature sweep with the same tuning budget. If it matches our failure-rate reduction, the wavelet machinery is unjustified, and we would report the finding as "late-stage attention range is the whole story."

Why not just use a convolutional backbone like DiCo? DiCo reaches FID 2.05 against DiT-XL/2's 2.27 at higher throughput. However, FID does not measure limb counts, and the analytic theory (An analytic theory of creativity in convolutional diffusion models) shows that pure convolutional diffusion is exactly the regime where late-stage patch mosaics arise. DiCo's authors also note that scenes with long-range dependencies may need hybrids. We will score DiCo samples with our structural failure protocol. If DiCo's failure rate is already below that of DiT plus our method, the backbone choice dominates and our contribution shrinks to a diagnostic.

Why not just generate a coarse blueprint first and then refine it? Cascades such as Imagen and da Silva Goncalves et al. do protect layout. Pan et al., however, note that cascades freeze early errors that later stages cannot repair. They also require training several models. Our method is a soft, per-token, inference-only version of the same idea. The comparison we can afford is not a retrained cascade but a hard anchor ($\Lambda_t = I$ for all tokens), which shows whether softness and gating matter.

Why not just add residual connections or normalization against over-smoothing? DiT already has residual connections and normalization. Residual connections provably slow GNN over-smoothing (Residual connections provably mitigate oversmoothing in graph neural networks). But our hypothesis is that the failure is drift in the coarse band, not collapse into it, so tools that prevent collapse target the wrong direction. The drift diagnostic in the first part of the program tests this directly: if failing samples show low coarse-band energy rather than high coarse-band drift, our hypothesis is wrong and normalization-style fixes become the right tool.

Proposed Experiments

The main benchmark is class-conditional ImageNet 256×256 with DiT-XL/2. The structure-sensitive subset consists of animal and human-containing classes, plus classes with regular grids or horizons. Standard ImageNet metrics are FID and Inception Score on 50k samples. The primary metric is a structural failure rate: blinded human raters, given fixed seeds, annotate extra or missing limbs and heads, detached parts, and broken horizons and grids, marking the failure region. A vision-language model judge is reported as a secondary metric and is calibrated against the human labels. For text-to-image, we use DrawBench, following Beyond Pixel Space, which shows qualitative fixes on prompts like "stack of cubes" but has no reported number on Lacuna. There we report rater preference and failure rate.

Three checks settle the idea. The first is mechanistic. On held-out samples, coarse-band drift between $t_c$ and the end of sampling must predict rater-labelled failure better than total feature drift or a global cosine-band drift does. In addition, on CIFAR10 and CelebA, the attention model's deviation from the local-score theory must sit mostly in the coarse bands. If drift does not predict failure, we stop.

The second is the intervention. Our method must lower the failure rate relative to plain DiT-XL/2 at matched FID. It must also beat temperature scaling, full-feature anchoring, and global-basis anchoring at equal tuning budget. Each ablation that matches the full method removes the corresponding part of the claim.

The third is localization. On samples where both our method and frequency-domain guidance fix the failure, our method should change pixels outside the annotated region less, measured as perceptual distance outside the region relative to the unmodified sample. This is the specific advantage that attention-graph wavelets are supposed to buy over a global basis.