LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers

Xuan Shen, Zhao Song, Yufa Zhou, Bo Chen, Yanyu Li, Yifan Gong, Kai Zhang, Hao Tan, Jason Kuen, Henghui Ding, Zhihao Shu, Wei Niu, Pu Zhao, Yanzhi Wang, Jiuxiang Gu

TL;DR

LazyDiT is a novel acceleration framework for Diffusion Transformers (DiTs) that dynamically skips redundant computations across consecutive denoising steps. By introducing lightweight linear layers trained with a specialized "lazy loss," the model learns when it can safely reuse cached outputs from previous steps, cutting computational costs by up to 50% with virtually no loss in image quality and enabling real-time generation on mobile devices.

What Problem Does This Paper Address?

While Diffusion Transformers (DiTs) have become the gold standard for high-fidelity image and video generation, their practical deployment is severely bottlenecked by slow inference speeds. Unlike traditional U-Net backbones, transformers scale beautifully but require evaluating billions of parameters at every single denoising step.

Prior attempts to accelerate diffusion models typically focus on reducing the total number of sampling steps or structurally pruning the network (e.g., removing layers permanently). However, the authors observe that permanently removing layers from DiTs causes a catastrophic drop in generation quality. This indicates that the redundancy in diffusion transformers isn't static within the architecture—it occurs dynamically between consecutive sampling steps. LazyDiT targets this temporal redundancy to accelerate inference without destroying the model's structural integrity.

Methods

The core insight of LazyDiT is grounded in mathematical analysis: the authors prove that the similarity between module outputs at consecutive timesteps has a remarkably high lower bound, and this similarity can be linearly approximated from the inputs using a Taylor expansion.

Building on this theory, the authors introduce a cache-based framework:

  1. Lazy Learning Layers: Lightweight linear layers are inserted right before every Multi-Head Self-Attention (MHSA) and pointwise Feedforward (FFN) module.
  2. Dynamic Skipping: At each denoising step, these linear layers predict a similarity score ($s$). If $s > 0.5$, the framework dynamically bypasses the heavy MHSA or FFN module entirely, instead pulling the cached result from the previous step.
  3. Lazy Loss Training: To train these layers, the original DiT weights are frozen. The new linear layers are trained for just 500 steps using a combination of standard diffusion loss and a novel "lazy loss" that explicitly encourages the model to maximize skipping where safe.
Figure 2: Overview framework. We skip the computation of MHSA or Feedforward by calling the previous step cache.

Datasets

The lazy learning layers were trained and evaluated using the standard ImageNet dataset. The authors validated their approach across multiple model configurations, including DiT-XL/2 (at $256\times256$ and $512\times512$ resolutions), LargeDiT-3B, and LargeDiT-7B.

Key Insights and Ideas

  • Temporal vs. Structural Redundancy: Redundancy in DiTs is highly dependent on the timestep. Bypassing layers dynamically on the fly works exceptionally well, whereas permanently pruning them ruins the model.
  • Predictable Similarity: Thanks to the scaling and shifting transformations inherent in DiT architectures, the difference between consecutive inputs remains tightly bounded, making it possible to accurately predict output similarity using a simple linear layer.
  • Layer-wise Behavior: Through ablation studies, the authors discovered that later MHSA layers and earlier FFN layers exhibit the highest redundancy, meaning they can be skipped more frequently without impacting the final image quality.

Key Results

LazyDiT consistently outperforms the standard DDIM sampler across various model sizes and resolutions.

  • Quality vs. Efficiency: As shown in the qualitative results, LazyDiT can skip 50% of the computations while generating images that are virtually indistinguishable from the full-computation baseline.
  • Quantitative Superiority: On DiT-XL/2 ($256\times256$), LazyDiT running for 50 steps with a 50% lazy ratio achieved a Fréchet Inception Distance (FID) of 2.70, outperforming a standard 25-step DDIM baseline (FID of 2.95) at a nearly identical computational cost (2.87 vs 2.86 TMACs).
  • Mobile Deployment: The authors successfully deployed LazyDiT on a smartphone powered by a Snapdragon 8 Gen 3 chip. It delivered significantly higher image quality than standard DDIM under identical latency constraints.
Figure 1: Image generated by DiT-XL/2 in 512×512 and 256×256 resolutions when lazily skipping 50% computation. The upper rows display results from original model and the lower rows showcase outcomes of our method. Our method generates distinct lighting effects for background and color compared to the baseline, as demonstrated in dog and marmot, respectively.

Limitations and Future Work

The primary limitation acknowledged by the authors is the minor computational overhead introduced by the lazy learning layers themselves. Although this overhead is minimized by projecting the hidden states down to a single dimension ($D_{out} = 1$), it still adds a small fraction of latency.

For future work, exploring how this lazy learning paradigm scales to text-to-video diffusion transformers (which are even more computationally demanding) represents a highly promising direction.

Summary updated

Abstract

Diffusion Transformers have emerged as the preeminent models for a wide array of generative tasks, demonstrating superior performance and efficacy across various applications. The promising results come at the cost of slow inference, as each denoising step requires running the whole transformer model with a large amount of parameters. In this paper, we show that performing the full computation of the model at each diffusion step is unnecessary, as some computations can be skipped by lazily reusing the results of previous steps. Furthermore, we show that the lower bound of similarity between outputs at consecutive steps is notably high, and this similarity can be linearly approximated using the inputs. To verify our demonstrations, we propose the \textbf{LazyDiT}, a lazy learning framework that efficiently leverages cached results from earlier steps to skip redundant computations. Specifically, we incorporate lazy learning layers into the model, effectively trained to maximize laziness, enabling dynamic skipping of redundant computations. Experimental results show that LazyDiT outperforms the DDIM sampler across multiple diffusion transformer models at various resolutions. Furthermore, we implement our method on mobile devices, achieving better performance than DDIM with similar latency. Code: https://github.com/shawnricecake/lazydit
BibTeX
@inproceedings{shen2025lazydit,
  author = {Xuan Shen and Zhao Song and Yufa Zhou and Bo Chen and Yanyu Li and Yifan Gong and Kai Zhang and Hao Tan and Jason Kuen and Henghui Ding and Zhihao Shu and Wei Niu and Pu Zhao and Yanzhi Wang and Jiuxiang Gu},
  title = {{LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i19.34248},
}
Figure