LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers

Xuan Shen, Zhao Song, Yufa Zhou, Bo Chen, Yanyu Li, Yifan Gong, Kai Zhang, Hao Tan, Jason Kuen, Henghui Ding, Zhihao Shu, Wei Niu, Pu Zhao, Yanzhi Wang, Jiuxiang Gu

Abstract

Diffusion Transformers have emerged as the preeminent models for a wide array of generative tasks, demonstrating superior performance and efficacy across various applications. The promising results come at the cost of slow inference, as each denoising step requires running the whole transformer model with a large amount of parameters. In this paper, we show that performing the full computation of the model at each diffusion step is unnecessary, as some computations can be skipped by lazily reusing the results of previous steps. Furthermore, we show that the lower bound of similarity between outputs at consecutive steps is notably high, and this similarity can be linearly approximated using the inputs. To verify our demonstrations, we propose the LazyDiT, a lazy learning framework that efficiently leverages cached results from earlier steps to skip redundant computations. Specifically, we incorporate lazy learning layers into the model, effectively trained to maximize laziness, enabling dynamic skipping of redundant computations. Experimental results show that LazyDiT outperforms the DDIM sampler across multiple diffusion transformer models at various resolutions. Furthermore, we implement our method on mobile devices, achieving better performance than DDIM with similar latency.

1. Introduction

Diffusion models, particularly those based on the Diffusion Transformer (DiT) architecture, have set new benchmarks in high-fidelity image and video generation. Unlike traditional U-Net-based models, DiTs scale effectively with increased data and parameter counts. However, this scalability introduces a significant computational bottleneck: the iterative denoising process requires multiple passes through a multi-billion parameter transformer, hindering real-time applications on edge devices.

Existing acceleration techniques, such as pruning or distillation, often compromise image quality. LazyDiT addresses this by identifying and exploiting temporal redundancy—the observation that internal outputs of transformer modules change minimally between consecutive denoising steps.

2. Methodology: The Lazy Learning Framework

LazyDiT introduces a dynamic skipping mechanism that operates at the module level. The core innovation is the "Lazy Learning" layer, a lightweight linear module inserted before each Multi-Head Self-Attention (MHSA) and Feedforward block.

Figure 2: The LazyDiT architecture featuring lazy learning layers that predict output similarity to enable dynamic skipping of MHSA and Feedforward modules.

As illustrated in Figure 2, the process involves:

  1. Similarity Prediction: The lazy layer predicts the cosine similarity ($s$) between the current module's potential output and the cached output from the previous timestep ($t-1$). The paper provides a theoretical foundation for this approach using Taylor expansion, proving that output similarity can be accurately approximated via a simple linear transformation of the input.
  2. Dynamic Skipping: If the predicted similarity exceeds a threshold (e.g., $s > 0.5$), the heavy computation of the MHSA or Feedforward block is skipped. Instead, the model reuses the cached result.
  3. Optimized Training: A specialized "Lazy Loss" is employed during a brief fine-tuning phase (approximately 10 minutes on 8 A100 GPUs) to maximize the skip ratio while preserving generative quality.

3. Experimental Results

The researchers evaluated LazyDiT on the ImageNet dataset at 256×256 and 512×512 resolutions across various model scales, including DiT-XL/2 and LargeDiT (3B and 7B parameters).

Figure 1: Visual comparison between standard DiT-XL/2 (play icon) and LazyDiT (fast-forward icon) at different resolutions, demonstrating high visual fidelity despite significant computational skipping.

Key findings include:

  • Efficiency vs. Quality: At a 50% lazy ratio, LazyDiT-XL/2 maintains an FID score comparable to the full baseline model. The authors also observed that different layers exhibit varying degrees of "laziness"; attention layers are more redundant in later denoising stages, while feedforward layers can often be skipped earlier.
  • Superiority over DDIM: When constrained to the same computational budget, LazyDiT consistently produces higher-quality images than the standard DDIM sampler running fewer steps. For example, at 512x512 resolution, LazyDiT with 50% skips achieved an FID of 3.94, outperforming DDIM at equivalent latency.
  • Mobile Performance: On a Snapdragon 8 Gen 3 processor, LazyDiT achieved significantly higher Inception Scores (IS) than DDIM for the same latency, making high-quality 10-step generation feasible on mobile hardware.

4. Conclusion and Future Work

LazyDiT demonstrates that temporal redundancy is a powerful lever for accelerating diffusion transformers. By shifting the focus from structural pruning to dynamic temporal skipping, LazyDiT achieves substantial speedups with minimal fine-tuning and negligible quality loss.

Future research will explore applying LazyDiT to video generation models, where temporal redundancy between frames is expected to be even more pronounced, potentially leading to even greater efficiency gains.

Summary updated

Abstract

Diffusion Transformers have emerged as the preeminent models for a wide array of generative tasks, demonstrating superior performance and efficacy across various applications. The promising results come at the cost of slow inference, as each denoising step requires running the whole transformer model with a large amount of parameters. In this paper, we show that performing the full computation of the model at each diffusion step is unnecessary, as some computations can be skipped by lazily reusing the results of previous steps. Furthermore, we show that the lower bound of similarity between outputs at consecutive steps is notably high, and this similarity can be linearly approximated using the inputs. To verify our demonstrations, we propose the \textbf{LazyDiT}, a lazy learning framework that efficiently leverages cached results from earlier steps to skip redundant computations. Specifically, we incorporate lazy learning layers into the model, effectively trained to maximize laziness, enabling dynamic skipping of redundant computations. Experimental results show that LazyDiT outperforms the DDIM sampler across multiple diffusion transformer models at various resolutions. Furthermore, we implement our method on mobile devices, achieving better performance than DDIM with similar latency.
BibTeX
@inproceedings{shen2025lazydit,
  author = {Xuan Shen and Zhao Song and Yufa Zhou and Bo Chen and Yanyu Li and Yifan Gong and Kai Zhang and Hao Tan and Jason Kuen and Henghui Ding and Zhihao Shu and Wei Niu and Pu Zhao and Yanzhi Wang and Jiuxiang Gu},
  title = {{LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i19.34248},
}
Figure