Abstract
Diffusion transformers (DiT) have set new benchmarks in generative modeling but suffer from high computational costs due to quadratic attention complexity and iterative sampling. We introduce TokenCache, a post-training acceleration framework that reuses redundant tokens across inference steps. By employing a lightweight Cache Predictor, adaptive block-level cache allocation, and a Two-Phase Round-Robin scheduling policy, TokenCache achieves significant speedups (2–3×) with minimal quality degradation. Our method is validated across various image and video generation tasks, outperforming existing state-of-the-art acceleration techniques.
Introduction
The transition from U-Net to Diffusion Transformer (DiT) architectures has significantly improved the scalability and quality of generative models like Sora and SD3. However, the computational demand of DiTs remains a barrier to real-time application. The primary bottlenecks are the multi-step denoising process and the self-attention mechanism, which scales quadratically with the number of tokens (image patches).
Existing acceleration methods often skip entire blocks or use static pruning, which fails to capture the spatial and temporal nuances of the denoising process. TokenCache addresses this by identifying and caching redundant tokens at a granular level, leveraging the observation that many image regions remain nearly static across consecutive timesteps.
Methodology
TokenCache optimizes computation across three dimensions: token-level, block-level, and temporal scheduling.
Token-Level Caching via Cache Predictor
The core of TokenCache is a lightweight Cache Predictor integrated into the DiT architecture. This module assigns importance scores to tokens; high-score tokens are recomputed to maintain detail, while low-score tokens are cached from the previous step. This selective computation bypasses the expensive attention mechanism for a large portion of the input.
Adaptive Block-Level Allocation
Not all transformer blocks contribute equally to the final output. TokenCache adaptively allocates a "cache ratio" to each block. Critical blocks receive more computation, while less impactful blocks rely more on cached tokens, optimizing the overall FLOPs-to-quality ratio.
Temporal Scheduling
To prevent error accumulation from continuous caching, the authors design a schedule that interleaves I-steps (Independent steps with full computation to "refresh" the cache) and P-steps (Prediction steps using the caching mechanism). This ensures that the model periodically re-anchors its generation process.
Experimental Results
We evaluated TokenCache on ImageNet (DiT-XL/2), MSCOCO (PixArt-α), and video generation (Open-Sora-Plan).
Efficiency and Quality
On DiT-XL/2 (256x256), TokenCache achieved a 3.01× speedup while maintaining an FID of 2.07. In contrast, the baseline DiT-XL/2 at a similar speedup level resulted in a significantly worse FID of 2.82. For high-resolution 512x512 synthesis, TokenCache reached a 1.63× speedup with an FID of 5.43, significantly outperforming competitors like ToCa (FID 13.50).
Video Generation
In video synthesis tasks using Open-Sora-Plan, TokenCache achieved the highest VBench score (72.36) and a 2.31× speedup. This demonstrates that the method effectively handles the temporal redundancy inherent in video data, where background elements often remain static across frames.
Conclusion
TokenCache provides a robust, input-adaptive framework for accelerating Diffusion Transformers. By intelligently managing token redundancy through a dedicated Cache Predictor and adaptive scheduling, it offers a superior trade-off between inference speed and generation quality. While it requires a lightweight training phase for the predictor, the resulting efficiency gains make high-fidelity diffusion models more viable for practical, real-world applications.
Summary updated