TL;DR
LazyDiT is a novel acceleration framework for Diffusion Transformers (DiTs) that dynamically skips redundant computations across consecutive denoising steps. By introducing lightweight linear layers trained with a specialized "lazy loss," the model learns when it can safely reuse cached outputs from previous steps, cutting computational costs by up to 50% with virtually no loss in image quality and enabling real-time generation on mobile devices.
What Problem Does This Paper Address?
While Diffusion Transformers (DiTs) have become the gold standard for high-fidelity image and video generation, their practical deployment is severely bottlenecked by slow inference speeds. Unlike traditional U-Net backbones, transformers scale beautifully but require evaluating billions of parameters at every single denoising step.
Prior attempts to accelerate diffusion models typically focus on reducing the total number of sampling steps or structurally pruning the network (e.g., removing layers permanently). However, the authors observe that permanently removing layers from DiTs causes a catastrophic drop in generation quality. This indicates that the redundancy in diffusion transformers isn't static within the architecture—it occurs dynamically between consecutive sampling steps. LazyDiT targets this temporal redundancy to accelerate inference without destroying the model's structural integrity.
Methods
The core insight of LazyDiT is grounded in mathematical analysis: the authors prove that the similarity between module outputs at consecutive timesteps has a remarkably high lower bound, and this similarity can be linearly approximated from the inputs using a Taylor expansion.
Building on this theory, the authors introduce a cache-based framework:
- Lazy Learning Layers: Lightweight linear layers are inserted right before every Multi-Head Self-Attention (MHSA) and pointwise Feedforward (FFN) module.
- Dynamic Skipping: At each denoising step, these linear layers predict a similarity score ($s$). If $s > 0.5$, the framework dynamically bypasses the heavy MHSA or FFN module entirely, instead pulling the cached result from the previous step.
- Lazy Loss Training: To train these layers, the original DiT weights are frozen. The new linear layers are trained for just 500 steps using a combination of standard diffusion loss and a novel "lazy loss" that explicitly encourages the model to maximize skipping where safe.
Datasets
The lazy learning layers were trained and evaluated using the standard ImageNet dataset. The authors validated their approach across multiple model configurations, including DiT-XL/2 (at $256\times256$ and $512\times512$ resolutions), LargeDiT-3B, and LargeDiT-7B.
Key Insights and Ideas
- Temporal vs. Structural Redundancy: Redundancy in DiTs is highly dependent on the timestep. Bypassing layers dynamically on the fly works exceptionally well, whereas permanently pruning them ruins the model.
- Predictable Similarity: Thanks to the scaling and shifting transformations inherent in DiT architectures, the difference between consecutive inputs remains tightly bounded, making it possible to accurately predict output similarity using a simple linear layer.
- Layer-wise Behavior: Through ablation studies, the authors discovered that later MHSA layers and earlier FFN layers exhibit the highest redundancy, meaning they can be skipped more frequently without impacting the final image quality.
Key Results
LazyDiT consistently outperforms the standard DDIM sampler across various model sizes and resolutions.
- Quality vs. Efficiency: As shown in the qualitative results, LazyDiT can skip 50% of the computations while generating images that are virtually indistinguishable from the full-computation baseline.
- Quantitative Superiority: On DiT-XL/2 ($256\times256$), LazyDiT running for 50 steps with a 50% lazy ratio achieved a Fréchet Inception Distance (FID) of 2.70, outperforming a standard 25-step DDIM baseline (FID of 2.95) at a nearly identical computational cost (2.87 vs 2.86 TMACs).
- Mobile Deployment: The authors successfully deployed LazyDiT on a smartphone powered by a Snapdragon 8 Gen 3 chip. It delivered significantly higher image quality than standard DDIM under identical latency constraints.
Limitations and Future Work
The primary limitation acknowledged by the authors is the minor computational overhead introduced by the lazy learning layers themselves. Although this overhead is minimized by projecting the hidden states down to a single dimension ($D_{out} = 1$), it still adds a small fraction of latency.
For future work, exploring how this lazy learning paradigm scales to text-to-video diffusion transformers (which are even more computationally demanding) represents a highly promising direction.
Summary updated