Sortblock: Similarity-Aware Feature Reuse for Diffusion Model

Hanqi Chen, Xu Zhang, Xiaoliu Guan, Lielin Jiang, Guanzhong Wang, Zeyu Chen, Yi Liu

Abstract

Diffusion Transformers (DiTs) have demonstrated remarkable generative capabilities, particularly benefiting from Transformer architectures that enhance visual and artistic fidelity. However, their inherently sequential denoising process results in high inference latency, limiting their deployment in real-time scenarios. Existing training-free acceleration approaches typically reuse intermediate features at fixed timesteps or layers, overlooking the evolving semantic focus across denoising stages and Transformer blocks. To address this, we propose Sortblock, a training-free inference acceleration framework that dynamically caches block-wise features based on their similarity across adjacent timesteps. By ranking the evolution of residuals, Sortblock adaptively determines a recomputation ratio, selectively skipping redundant computations while preserving generation quality. Furthermore, we incorporate a lightweight linear prediction mechanism to reduce accumulated errors in skipped blocks. Extensive experiments across various tasks and DiT architectures demonstrate that Sortblock achieves over 2× inference speedup with minimal degradation in output quality, offering an effective and generalizable solution for accelerating diffusion-based generative models.

Introduction

Diffusion Transformers (DiTs) have established themselves as the state-of-the-art for high-fidelity image and video generation, powering models such as Flux and Sora. However, the iterative nature of the denoising process is computationally intensive and slow. While "training-free" acceleration methods exist, they often rely on static caching strategies—reusing features from previous timesteps at fixed intervals or specific layers.

Such approaches are inherently inefficient because the denoising process is non-uniform: early stages typically establish global structure, while later stages refine fine-grained textures. A static caching strategy often leads to structural distortions or loss of detail. Sortblock addresses these limitations by making the caching process adaptive and block-aware, dynamically identifying and skipping redundant computations based on real-time similarity metrics.

The Sortblock Framework

Sortblock optimizes DiT inference through three primary mechanisms designed to make feature reuse smarter and more accurate:

  1. Similarity-Aware Ranking: Instead of skipping blocks on a fixed schedule, Sortblock calculates the cosine similarity of the "residuals" (the change between input and output) for each block across adjacent timesteps. It ranks these blocks and recomputes only those that have changed the most (lowest similarity), while skipping the redundant ones.
  2. Linear Prediction Mechanism: Simply copying a feature from a previous step leads to accumulated errors. Sortblock employs a lightweight linear extrapolation to predict the feature's state at the current step based on its previous trajectory. This first-order approximation significantly improves accuracy with negligible computational overhead.
  3. Dynamic Recomputation Ratio (ρ): Redundancy is highest in the middle of the denoising process. Sortblock uses a 5th-degree polynomial curve to adaptively adjust the ratio of recomputed blocks at each stage, ensuring high fidelity during the critical early and late stages of generation.
Visualization of Flux blocks and L1 differences across dual and single-stream blocks.

Key Insights and Analysis

The development of Sortblock is grounded in two critical observations regarding DiT behavior:

  • The "70% Rule": Empirical analysis across multiple models (Flux.1-dev, Wan 2.1, HunyuanVideo) reveals that approximately 70% of the diffusion process—primarily the middle steps—exhibits very low variation in feature outputs. This temporal redundancy provides a significant opportunity for computational savings.
  • Block Sensitivity: Not all Transformer blocks contribute equally at every step. Shallow blocks (near the input) capture global layouts, while deeper blocks refine details. Sortblock’s ability to treat these blocks differently based on real-time similarity allows it to maintain quality where static methods fail.
L1 difference across the diffusion process for various models, highlighting temporal redundancy in the middle stages.

Experimental Results

Sortblock consistently outperformed existing methods like PAB, T-GATE, and TeaCache in the speed-vs-quality trade-off across leading-edge architectures:

  • Flux.1-dev: Achieved a 2.39× speedup, reducing latency from 23.23s to 9.7s per image, while maintaining higher Image Reward and SSIM scores than competitors.
  • Wan2.1 (Video): Delivered a 2.0× speedup (38s vs. 86s) with a VBench score of 79.47%, nearly identical to the 79.55% baseline.
  • HunyuanVideo: Achieved a 2.4× speedup while actually improving PSNR and SSIM compared to other training-free acceleration baselines.

Conclusion and Future Work

Sortblock provides a robust, training-free solution for accelerating Diffusion Transformers by leveraging inherent temporal and structural redundancies. By combining similarity-aware ranking with linear prediction, it achieves over 2× speedup with negligible impact on output quality. Future research will focus on automating hyperparameter tuning for arbitrary models and optimizing the framework for consumer-grade hardware.

Summary updated

Abstract

Diffusion Transformers (DiTs) have demonstrated remarkable generative capabilities, particularly benefiting from Transformer architectures that enhance visual and artistic fidelity. However, their inherently sequential denoising process results in high inference latency, limiting their deployment in real-time scenarios. Existing training-free acceleration approaches typically reuse intermediate features at fixed timesteps or layers, overlooking the evolving semantic focus across denoising stages and Transformer blocks.To address this, we propose Sortblock, a training-free inference acceleration framework that dynamically caches block-wise features based on their similarity across adjacent timesteps. By ranking the evolution of residuals, Sortblock adaptively determines a recomputation ratio, selectively skipping redundant computations while preserving generation quality. Furthermore, we incorporate a lightweight linear prediction mechanism to reduce accumulated errors in skipped blocks.Extensive experiments across various tasks and DiT architectures demonstrate that Sortblock achieves over 2$\times$ inference speedup with minimal degradation in output quality, offering an effective and generalizable solution for accelerating diffusion-based generative models.
BibTeX
@inproceedings{chen2026sortblock,
  author = {Hanqi Chen and Xu Zhang and Xiaoliu Guan and Lielin Jiang and Guanzhong Wang and Zeyu Chen and Yi Liu},
  title = {{Sortblock: Similarity-Aware Feature Reuse for Diffusion Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i4.37276},
}
Figure