Phase-Wise Linear Folding for Diffusion Transformers on Edge Devices

Diffusion models generate an image by running the same large network tens of times, each pass removing a little noise. Because consecutive passes see nearly identical inputs, a popular speedup is feature caching: skip some blocks of the network on some steps and reuse the outputs saved from an earlier step. Caching works on server GPUs. On phones and embedded boards it runs into a different constraint. The saved activations are large tensors that must be written to and read back from slow shared memory, and the skip decisions often depend on the input, which static edge compilers handle poorly. This proposal replaces stale reuse with something that costs no activation memory. For each stretch of the sampling trajectory, we fit a small linear map that stands in for a run of skipped blocks, then multiply it into the weights of the block just before them. The deployed model becomes a handful of fixed, shallower networks, one per stretch of timesteps. Each has a static graph, and none needs an activation buffer.

The bet is that temporal redundancy and linearizability are the same phenomenon viewed from two sides. A block whose output barely changes between adjacent steps is also a block whose input-to-output behavior, restricted to those steps, is close to linear. If that holds, a linear map fitted per phase should get past the accuracy cliff that training-free linear layer replacement hits when one map must cover all timesteps. It should also beat caching on the metric edge hardware cares about, wall-clock latency under a memory-bandwidth limit.

Core Claim

Two lines of work sit on either side of this idea without touching. Training-free block linearization, as in ReplaceMe, picks a contiguous run of Transformer layers, fits one linear map that reproduces their net effect on a calibration set, and folds it into the preceding layer's MLP output projection (the "down-projection"), so the deployed model has no extra parameters. The file reports it keeps 92.5% of LLaMA-2-7B accuracy at 25% depth reduction and degrades sharply beyond about 25 to 30%. Temporal caching methods for Diffusion Transformers (DiTs, diffusion models built from Transformer blocks) such as SmoothCache, BWCache, Δ-DiT and BudCache exploit the fact that block features follow a U-shaped trajectory: they change fast early, stabilize in the middle, and change fast again late. None of these caching methods modifies weights. All pay for reuse with stored activations.

Our claim is that a DiT block's function becomes much more linear when restricted to one phase of the trajectory. There is a mechanistic reason to expect this. DiT blocks receive the timestep through adaptive normalization, per-channel scales, shifts and output gates computed from $t$. Within a phase these are nearly constant, so the time-varying part of the block's behavior disappears. Concretely, for skipped blocks $b+1,\dots,b+n$ and phase $P_k$ (a contiguous set of timesteps), we fit

$$T_{b,k} = \arg\min_T \sum_{t \in P_k} \sum_{x} \big\| \,(h_{b+n}(x,t) - h_b(x,t)) - T\, m_b(x,t)\,\big\|^2,$$

where $h_b$ is the residual-stream hidden state after block $b$, $m_b$ is block $b$'s gated MLP output, and $x$ ranges over calibration samples. This is ordinary least squares with a closed form. Because the gate is roughly constant within $P_k$, $(I + T_{b,k})$ folds exactly into block $b$'s down-projection and gate, which gives a per-phase weight variant with the same compute cost as the original down-projection and no stored activations. The testable prediction is that the relative fit residual of $T_{b,k}$ tracks the U-shape. It should be small in middle phases and large at early and late steps, and markedly smaller than the residual of a single timestep-agnostic map fitted over all $t$.

The memory argument is simple and checkable. A cached block output is an $N \times d$ tensor per sample, where $N$ is the token count and $d$ the hidden width, and classifier-free guidance doubles it. The folded map adds nothing at runtime, and storing $K$ phase variants of one down-projection costs weight storage that is read sequentially and can be quantized. For DiT-XL/2 ($d = 1152$), $N$ is 256 at 256px and 1024 at 512px. Higher-resolution text-to-image models push $N$ well past $d$. The advantage of folding over caching should therefore grow with $N/d$ on bandwidth-limited devices and be small or absent on server GPUs. That is a prediction the edge experiments can falsify.

Research Program

The work has three parts. The first is diagnostic. On DiT-XL/2 (ImageNet 256 and 512) and PixArt-α (MS-COCO prompts), we record trajectories for a few hundred calibration samples. For every candidate run of $n \in \{1,2,3,4\}$ consecutive blocks and every phase partition, we measure the per-phase linear-fit residual next to the step-to-step relative $L_1$ change that SmoothCache and BWCache use as their skip criterion. If the fit residual does not rise and fall with the caching criterion, the central hypothesis fails and we report that. We will also look at Δ-DiT's finding that deep blocks shape low-frequency layout early and shallow blocks shape detail late. It motivates the design, because it suggests which blocks will linearize in which phase. It may also constrain it: if shallow blocks at late steps turn out to be strongly nonlinear, those regions must stay dense.

The second part is the scheduler. The choice of which block runs to fold in which phase, under a latency budget, is a discrete optimization. We reuse two existing techniques. From LayerMerge we take dynamic programming over measured per-layer latency tables rather than FLOP counts. LayerMerge applied that DP to static CNNs and a DDPM U-Net, running the same pruned graph at every step. From BudCache we take its budget-first stance: produce a deterministic static schedule for a target cost. BudCache searched only which whole steps to cache and ran each evaluated step at full size. Our search space is (block run × phase), the latency tables are measured on the target edge device, and the quality term is a trajectory-level error on held-out calibration samples, so errors that compound across steps are penalized. The output is $K$ compiled static graphs and a fixed switch schedule, which is what edge toolchains require.

The third part is edge validation. We compile the resulting models with TensorRT on an NVIDIA Jetson Orin, and with ONNX Runtime on an ARM mobile CPU. Where feasible, we also target a phone NPU through its vendor static-graph toolchain. We report latency, peak memory, and energy per image, and we check for operator fallback. Prior edge work in the file, including FDFlowNet and DepthShrinker, is what motivates measuring rather than estimating. FDFlowNet documents CPU fallback erasing theoretical savings. DepthShrinker's Jetson TX2 results show that folded dense layers can run far faster than their FLOP count suggests. Folding produces only standard dense matmuls, so we expect no fallback. Caching baselines with runtime skip decisions may need graph breaks, and we will report what actually happens rather than assume it.

Baselines

The naive reference is the unmodified DiT running every step with a static graph at the same sampler and step count. All speedups and quality deltas are reported against it on the same device.

The strongest caching competitor on quality is Δ-DiT. It caches differences between features rather than raw features and schedules deep versus shallow blocks by phase. The file reports that on DiT-XL, 20-step sampling, it reaches 1.6× speedup with FID going from 15.893 to 13.289. On PixArt-α it reaches 1.6× with FID going from 39.002 to 35.882. It is the closest in spirit, since it is also phase-aware and training-free, so it isolates whether weight folding beats activation reuse. SmoothCache, reported at up to 1.71× on DiT-XL, is the simplest calibrated static-schedule cache and is the natural baseline for edge compilation, because its schedule is fixed after calibration. BudCache is the strongest budget-constrained baseline. The file reports LPIPS 0.1759 at 2.51× on FLUX.1-dev against TeaCache's 0.3218; that model is too large for our edge targets, so we rerun its search on DiT-XL and PixArt-α. Every caching baseline is run with its official schedule procedure, the same calibration samples we use (count matched), the same sampler and step count, and compiled on the same device. Where a dynamic threshold prevents static compilation, we also give it the most favorable static approximation of its schedule.

The strongest static-compression competitor is TinyFusion. It learns which DiT layers to drop, optimizing for recoverability after fine-tuning, and reaches FID 2.86 against 2.27 for the full DiT-XL at 2× speedup, using under 7% of pre-training compute. It uses far more compute than our training-free method, so we compare at two points. First, TinyFusion's published 50%-depth model against our method at matched on-device latency. Second, both methods at a matched small fine-tuning budget, where we allow LoRA healing of our folded model with TinyFusion's recipe.

Two ablations isolate the key ingredient. The first is a timestep-agnostic fold: one $T_b$ fitted over all timesteps, which is ReplaceMe transplanted to DiTs. The second is per-phase folding with the per-phase gate folding disabled. Beating Δ-DiT and SmoothCache means lower FID at equal measured edge latency, or lower edge latency at equal FID, on at least DiT-XL/2 512 and PixArt-α. Beating the timestep-agnostic fold at depth reductions beyond 30% is the result that shows phase conditioning is what matters.

Why not just…?

Why not just stack TinyFusion with a cache like BudCache? The two are plausibly orthogonal, and this is the most serious threat. If "prune, then cache" matches our Pareto curve, then per-phase folding adds nothing beyond engineering. We treat it as a real baseline, not a strawman. The composition inherits TinyFusion's fine-tuning cost and the cache's activation buffers, so we expect it to lose specifically on edge memory and latency at high token counts, while perhaps matching or winning on server GPUs. We also run the reverse composition, our folding on top of TinyFusion's shallow model, because the honest outcome may be that the methods compose rather than compete.

Why not just use caching with a static schedule, since activation memory is small? At DiT-XL/2 256px, $N < d$, and caching's memory cost is genuinely modest; we expect little difference there. The crossover analysis across 256px, 512px and PixArt-α's higher-resolution configurations is the experiment that settles it. If folding shows no latency advantage over static caching even when $N/d \gg 1$ on the Jetson and mobile CPU, the edge argument is wrong and the contribution shrinks to a quality comparison.

Why not just fit one linear map, as ReplaceMe does? ReplaceMe's reported collapse past 25 to 30% depth on LLMs is exactly what phase conditioning is meant to fix, but DiTs might be more linear than LLMs to begin with. The timestep-agnostic ablation answers this directly. If it matches per-phase folding, the phase machinery is unnecessary and we will say so.

Why not just use fewer sampling steps? BudCache reports that temporal caching still helps on distilled 8-step models, which suggests redundancy survives step reduction. Our method depends on middle-phase stability, though, and Δ-DiT's 4-step result of only 1.12× hints that redundancy thins at few steps. We therefore test at 50, 20 and 8 steps and expect gains to shrink as step count falls.

Proposed Experiments

The first decisive check is the diagnostic. Per-phase linear-fit residual must correlate with the step-to-step change profile across blocks and timesteps, and must fall well below the timestep-agnostic residual in the middle phases. If it does not, we stop and publish the negative result.

The second is quality. We measure FID-50K on ImageNet 256 and 512 with DiT-XL/2, and FID and CLIP score on MS-COCO2017 with PixArt-α, plotted against measured on-device latency for folding, the caching baselines, TinyFusion, and the compositions. Because LLM-Streamline warns that aggregate metrics can hide erratic behavior after compression, we also report per-sample LPIPS and PSNR against the uncompressed model's output from the same noise seed, which exposes trajectory drift that FID can average away.

The third is the edge crossover. On Jetson Orin and an ARM CPU, we plot latency and peak memory of folding versus the best cache at matched FID as $N/d$ grows, and report operator fallbacks per method. A server GPU (A100) run with the same models shows whether the advantage is edge-specific, as predicted.

The last check probes where the method breaks. We push depth reduction past 30%, 40% and 50% and record where per-phase folding hits its own cliff, which blocks and phases resist linearization, and whether a small LoRA healing budget moves that cliff further than it moves TinyFusion's.