A diffusion model is a generator that learns to turn random noise into images by repeatedly removing a little noise at a time. During training, each example is corrupted to a randomly chosen noise level and the network learns to undo that corruption. At high noise levels, most of the fine detail in the image has been drowned out. Only coarse layout and color survive in the input. Yet a diffusion transformer (DiT), which is a transformer that processes the image as a grid of patch tokens, spends the same attention budget at every noise level: each token compares itself against every other token at full spatial resolution. This proposal asks the attention layers to look at a coarser grid when the input carries only coarse information. Every query token stays, so the output keeps full resolution. The keys and values that tokens attend to are spatially pooled by an amount set by the noise level. The model sees full-resolution context only at the low-noise steps, where fine detail actually exists.
Stated in two sentences: the amount of image detail that survives noising follows a predictable curve, and attention context finer than that curve carries mostly noise. We therefore train diffusion transformers whose key/value grid resolution tracks the noise schedule. This uses ordinary dense attention kernels, so the theoretical savings show up as real wall-clock savings. The payoff grows with resolution, because attention cost grows with the square of the token count. At 1024 and 2048 pixels, attention stops being a minor cost and becomes the dominant one.
Core Claim
Our claim is that a fixed, analytically motivated schedule can remove most attention compute at high noise levels during training from scratch with no measurable loss in sample quality. The schedule maps the noise level $t$ to a key/value pooling factor. It needs no learned router, no teacher model, no token dropping, and no custom sparse kernel. It should also beat the same compute saving spent timestep-agnostically, whether as uniform key/value pooling or as uniform token dropping.
Three things distinguish this from the closest work in the file. DiffCR learns layer- and timestep-dependent token compression ratios with a router. It is applied as fine-tuning of a pretrained PixArt-Σ, and its differentiable ratio binning doubles the forward pass. Bidirectional Sparse Attention prunes queries and key/value blocks during video DiT training. Its thresholds come from attention-score statistics that ignore the noise level, it needs custom Triton kernels, and it fills pruned query positions by copying one critical token. TREAD routes a random subset of tokens past intermediate layers, with the same route at every timestep. Our method differs in three ways. It never removes a query token. It sets its budget from the diffusion noise schedule rather than from learned or statistical scores. And it runs entirely on standard dense attention over a shorter key/value sequence. Shortening the key/value sequence by pooling is a known efficient-attention device. The new part is tying the pooling factor to the signal-to-noise ratio of the diffusion input and testing whether that tie matters during training.
Why the schedule has a predictable shape
Natural images have a power spectrum that falls roughly as $S(f) \approx A f^{-2}$, where $f$ is spatial frequency and $A$ a constant. A noised input is $x_t = \alpha_t x_0 + \sigma_t \epsilon$, where $x_0$ is the clean image, $\epsilon$ is white Gaussian noise, and $\alpha_t, \sigma_t$ are the schedule coefficients. Frequency $f$ is signal-dominated only while $\alpha_t^2 S(f) > \sigma_t^2$. That gives a cutoff
$$ f_c(t) = \sqrt{A}\,\frac{\alpha_t}{\sigma_t} = \sqrt{A \cdot \mathrm{SNR}(t)}. $$
Above $f_c(t)$ the input mostly contains noise, and the optimal denoiser mostly shrinks it. A key/value grid with side length proportional to $f_c(t)$ therefore loses little of the information present at that noise level. Attention cost scales with query count times key count, so its cost relative to full attention scales roughly as $\min(1, \mathrm{SNR}(t)/\mathrm{SNR}^\ast)$. Here $\mathrm{SNR}^\ast$ is the level at which the cutoff reaches the full token grid.
This argument is only a prior, for two reasons. Latent diffusion operates on autoencoder latents, whose spectrum is not exactly $f^{-2}$. Deep layers also hold features that are no longer image-like. The first stage of the work therefore measures the per-layer cutoff empirically and treats the formula as the hypothesis to check.
The motivating evidence fits this picture. DiffCR found that early, high-noise timesteps tolerate aggressive token compression while late timesteps need detail. The attention-sink analysis found that at early, high-noise steps over 99.9% of sink attention mass sits on text-conditioning tokens, not image tokens. That is a sign that fine image-to-image attention matters little there. Adaptive Non-Uniform Timestep Sampling constrains the design from the other side: low-noise timesteps have the worst gradient variance and optimization difficulty. Our schedule deliberately leaves those steps at full attention and saves only where the optimization is easy.
Research Program
We build the model in the standard DiT codebase (Scalable Diffusion Models with Transformers). In each self-attention layer, the key and value token grids are average-pooled by a factor $r(t, \ell) \in \{1, 2, 4, 8\}$ per side before attention, where $\ell$ is the layer index. Queries remain at full resolution. Text tokens used in cross-attention are never pooled, because Hard-Coded Gaussian Attention shows that fixing or simplifying cross-attention collapses quality even when self-attention tolerates it. Pooled keys still cover the whole image. This avoids the long-range failure that the same paper reports for local fixed patterns, and it avoids the oversmoothing that DyDiLA documents for linear attention, because softmax is retained.
A mixed batch would contain different key/value lengths. We handle this in two ways and compare them. The first stratifies timestep sampling so that each micro-batch shares one pooling bucket. The second uses variable-length dense attention. We check that stratification does not change the effective timestep distribution or gradient noise.
The work has three stages.
The first stage is diagnostic and needs no training. On a pretrained DiT-XL/2 at 512 pixels, we replace full keys and values with pooled ones for each layer and each noise level, one at a time. We measure the relative error of the attention output and the change in the model's denoising loss. The result is an empirical cutoff map $r^\ast(t, \ell)$, which we compare with the $\sqrt{\mathrm{SNR}}$ prediction.
The second stage trains from scratch on ImageNet at 256 and 512 pixels, with the schedule either fit to $r^\ast$ or set by the formula. The result is compared at matched wall-clock and matched GPU-hours.
The third stage moves to the regime where attention dominates: 1024- and 2048-pixel text-to-image training in a PixArt-Σ-style setup, the backbone that LiT and DiffCR used.
A simple FLOP count sets expectations honestly. Take a DiT block of width $d$ with $N$ tokens. The attention score and value products cost about $4N^2 d$, against roughly $24Nd^2$ for the projections and MLP, a ratio of $N/6d$. For XL width ($d = 1152$), attention is about 4% of block compute at 256 pixels ($N = 256$) and about 13% at 512 ($N = 1024$). At 1024 pixels ($N = 4096$) it is about 37%, and at 2048 pixels it is about 70%.
So we predict negligible end-to-end gains at 256 pixels. That setting serves as a quality control. We predict modest gains at 512 and substantial gains at 1024 and above. Because the method leaves the MLP and query count untouched, it composes with TREAD, which cuts token count in middle layers. The combination is one of the planned configurations.
Baselines
The naive reference is a vanilla DiT with FlashAttention, trained at the same resolution, batch size, optimizer, and schedule. The file reports DiT-XL/2 at FID 2.27 on ImageNet-256 and FID 3.04 on ImageNet-512 (Scalable Diffusion Models with Transformers), both after long training. We reproduce it at our budget so every comparison uses equal compute.
The strongest existing training accelerator is TREAD. It reports FID 2.09 guided and 3.93 unguided on ImageNet-256, and 3.03 iterations per second against 1.86 for DiT. It is the strongest competitor because it is architecture-agnostic, needs no unmasking stage, and already reduces attention cost by shortening sequences. We found no reported result on Lacuna for TREAD at 512 pixels or at text-to-image resolutions of 1024 and above, so we run it ourselves with its published route. We also give it the same hyperparameter search budget as our schedule. MaskDiT is a second accelerator baseline, reporting FID 2.28 guided on ImageNet-256 in about 30% of DiT training time. We run it with its unmasking phase counted in the compute budget.
The high-resolution text-to-image comparison uses LiT, a linear-attention alternative. It reports GenEval comparable to PixArt-Σ and about 55% lower latency at 2K. We found no reported GenEval number for it on Lacuna. LiT depends on a pretrained teacher, so we report it in two forms: as published, with the teacher cost stated separately, and trained from scratch without inheritance.
Two ablations isolate the key ingredient. The first is uniform pooling: a constant key/value pooling factor chosen to match our schedule's average attention FLOPs across timesteps. This tests whether the dependence on noise level matters, or just the pooling. The second is the reversed schedule, with coarse keys at low noise and fine keys at high noise, as a sanity check that the direction matters. We also run a timestep-scheduled TREAD. It drops more tokens at high noise under the same budget, which tests whether our advantage comes from keeping queries or merely from scheduling by noise level.
Beating a baseline means one of two things. Either we reach the baseline's FID, or its GenEval score at high resolution, in at least 20% less measured wall-clock time on the same hardware. Or we reach a lower FID at equal wall-clock, with the gap exceeding the seed-to-seed spread across three seeds.
Why not just…?
Why not just use FlashAttention and accept quadratic cost? FlashAttention reduces memory traffic but not the $N^2$ arithmetic. Rapid Diffusion reports that even after FlashAttention and compiler fusion, cross-attention alone remains over 43% of runtime. Our method is built on FlashAttention: it hands FlashAttention a shorter key sequence. At 256 pixels this alternative really is sufficient, and our own FLOP count says so. We report that outcome rather than hide it.
Why not just use TREAD or MaskDiT, which already shorten sequences? This is the most serious threat. Both remove tokens timestep-agnostically, and MaskDiT must repair the lost information with an unmasking stage. They reduce MLP cost, which our method does not. So at 256 and 512 pixels TREAD may well win outright on wall-clock. The experiments settle this in three places. The head-to-head at 1024 and above shows where attention dominates. The timestep-scheduled TREAD ablation tests whether keeping queries matters. The TREAD-plus-schedule combination tests whether the two savings add. If scheduled TREAD matches our method everywhere, the contribution shrinks to "schedule token dropping by SNR," and we would report it that way.
Why not just learn the compression per layer and timestep, as DiffCR does? A learned router can in principle find a better allocation. But DiffCR doubles the forward pass during its adaptation, and its ratios were learned on a pretrained model, not during pretraining. If our stage-one cutoff map closely matches the analytic curve, a learned router has little room to improve. We test this by comparing our fixed schedule against the empirically fit per-layer map. The gap between them bounds what learning could add.
Why not just train at low resolution first and then fine-tune at high resolution? Progressive resolution also saves attention compute, and it is a natural competitor for the 1024-pixel experiments. It saves by training the whole network, MLP included, at low resolution for a period. Our method saves only on high-noise attention, within full-resolution training. We include a progressive-resolution run at equal GPU-hours. The two can also be combined, since our schedule applies inside each resolution stage.
Decisive Checks
Three results would kill or confirm the idea.
The first comes from stage one. If pooling keys and values by 4× at high noise levels changes the pretrained DiT-XL/2's denoising loss about as much as it does at low noise levels, the premise is false, and we stop before any training.
The second is at 512 pixels on ImageNet. The scheduled model must match the vanilla DiT's FID curve at equal steps within seed variance, and it must beat the uniform-pooling ablation at equal attention FLOPs. This comparison is the direct test of whether the noise level is the right thing to schedule on.
The third is at 1024 and 2048 pixels in text-to-image training. Measured on the same GPUs, the scheduled model must reach the vanilla model's COCO FID and GenEval score in clearly less wall-clock time. It must also do at least as well as TREAD and from-scratch LiT on the same budget.
A secondary check covers sampling. The model trained with the schedule can also be sampled with it. We measure whether that gives an inference speedup for free, and we compare it with TokenCache, which reports a 3.01× speedup at FID 2.07 on DiT-XL/2 at 256 pixels. If the schedule helps at both training and sampling time, it offers something that inference-only caching methods cannot.
References
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers
- Bidirectional Sparse Attention for Faster Video Diffusion Training
- TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training
- Attention Sinks in Diffusion Transformers: A Causal Analysis
- Adaptive Non-Uniform Timestep Sampling for Accelerating Diffusion Model Training
- Scalable Diffusion Models with Transformers
- Hard-Coded Gaussian Attention for Neural Machine Translation
- Dynamic Differential Linear Attention: Enhancing Linear Diffusion Transformer for High-Quality Image Generation
- LiT: Delving into a Simple Linear Diffusion Transformer for Image Generation
- Fast Training of Diffusion Models with Masked Transformers
- Rapid Diffusion: Building Domain-Specific Text-to-Image Synthesizers with Fast Inference Speed
- Token Caching for Diffusion Transformer Acceleration