Token Caching for Diffusion Transformer Acceleration

Jinming Lou, Wenyang Luo, Yufan Liu, Bing Li, Xinmiao Ding, Weiming Hu, Jiajiong Cao, Yuming Li, Chenguang Ma

Abstract

Diffusion transformers (DiT) have set new benchmarks in generative modeling but suffer from high computational costs due to quadratic attention complexity and iterative sampling. We introduce TokenCache, a post-training acceleration framework that reuses redundant tokens across inference steps. By employing a lightweight Cache Predictor, adaptive block-level cache allocation, and a Two-Phase Round-Robin scheduling policy, TokenCache achieves significant speedups (2–3×) with minimal quality degradation. Our method is validated across various image and video generation tasks, outperforming existing state-of-the-art acceleration techniques.

Introduction

The transition from U-Net to Diffusion Transformer (DiT) architectures has significantly improved the scalability and quality of generative models like Sora and SD3. However, the computational demand of DiTs remains a barrier to real-time application. The primary bottlenecks are the multi-step denoising process and the self-attention mechanism, which scales quadratically with the number of tokens (image patches).

Existing acceleration methods often skip entire blocks or use static pruning, which fails to capture the spatial and temporal nuances of the denoising process. TokenCache addresses this by identifying and caching redundant tokens at a granular level, leveraging the observation that many image regions remain nearly static across consecutive timesteps.

Methodology

TokenCache optimizes computation across three dimensions: token-level, block-level, and temporal scheduling.

Token-Level Caching via Cache Predictor

The core of TokenCache is a lightweight Cache Predictor integrated into the DiT architecture. This module assigns importance scores to tokens; high-score tokens are recomputed to maintain detail, while low-score tokens are cached from the previous step. This selective computation bypasses the expensive attention mechanism for a large portion of the input.

Detailed TokenCache architecture showing the I-Step (Full) and P-Step (Cache) mechanisms alongside the Cache Predictor.

Adaptive Block-Level Allocation

Not all transformer blocks contribute equally to the final output. TokenCache adaptively allocates a "cache ratio" to each block. Critical blocks receive more computation, while less impactful blocks rely more on cached tokens, optimizing the overall FLOPs-to-quality ratio.

Temporal Scheduling

To prevent error accumulation from continuous caching, the authors design a schedule that interleaves I-steps (Independent steps with full computation to "refresh" the cache) and P-steps (Prediction steps using the caching mechanism). This ensures that the model periodically re-anchors its generation process.

Experimental Results

We evaluated TokenCache on ImageNet (DiT-XL/2), MSCOCO (PixArt-α), and video generation (Open-Sora-Plan).

Comparison of conceptual cost reduction and empirical FLOPs vs. FID performance against state-of-the-art baselines.

Efficiency and Quality

On DiT-XL/2 (256x256), TokenCache achieved a 3.01× speedup while maintaining an FID of 2.07. In contrast, the baseline DiT-XL/2 at a similar speedup level resulted in a significantly worse FID of 2.82. For high-resolution 512x512 synthesis, TokenCache reached a 1.63× speedup with an FID of 5.43, significantly outperforming competitors like ToCa (FID 13.50).

Video Generation

In video synthesis tasks using Open-Sora-Plan, TokenCache achieved the highest VBench score (72.36) and a 2.31× speedup. This demonstrates that the method effectively handles the temporal redundancy inherent in video data, where background elements often remain static across frames.

Conclusion

TokenCache provides a robust, input-adaptive framework for accelerating Diffusion Transformers. By intelligently managing token redundancy through a dedicated Cache Predictor and adaptive scheduling, it offers a superior trade-off between inference speed and generation quality. While it requires a lightweight training phase for the predictor, the resulting efficiency gains make high-fidelity diffusion models more viable for practical, real-world applications.

Summary updated

Abstract

Diffusion transformers have gained substantial interest in diffusion generative modeling due to their outstanding performance. However, their high computational cost, arising from the quadratic computational complexity of attention mechanisms and multi-step inference, presents a significant bottleneck. To address this challenge, we propose TokenCache, a novel post-training acceleration method that leverages the token-based multi-block architecture of transformers to reduce redundant computations among tokens across inference steps. TokenCache specifically addresses three critical questions in the context of diffusion transformers: (1) which tokens should be pruned to eliminate redundancy, (2) which blocks should be targeted for efficient pruning, and (3) at which time steps caching should be applied to balance speed and quality. In response to these challenges, TokenCache introduces a Cache Predictor that assigns importance scores to tokens, enabling selective pruning without compromising model performance. Furthermore, we propose an adaptive block selection strategy to focus on blocks with minimal impact on the network's output, along with a Two-Phase Round-Robin (TPRR) scheduling policy to optimize caching intervals throughout the denoising process. Experimental results across various models demonstrate that TokenCache achieves an effective trade-off between generation quality and inference speed for diffusion transformers. Our code will be publicly available.
BibTeX
@misc{lou2024token,
  author = {Jinming Lou and Wenyang Luo and Yufan Liu and Bing Li and Xinmiao Ding and Weiming Hu and Jiajiong Cao and Yuming Li and Chenguang Ma},
  title = {{Token Caching for Diffusion Transformer Acceleration}},
  year = {2024},
  eprint = {2409.18523},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.18523},
}
Figure