Sortblock: Similarity-Aware Feature Reuse for Diffusion Model

Hanqi Chen, Xu Zhang, Xiaoliu Guan, Lielin Jiang, Guanzhong Wang, Zeyu Chen, Yi Liu

TL;DR

SortBlock is a training-free acceleration framework for Diffusion Transformers (DiTs) that achieves over 2× inference speedup with negligible loss in image and video quality. By dynamically identifying and skipping redundant computations in specific transformer blocks based on their similarity across timesteps, it bypasses the need for expensive retraining or distillation.

What Problem Does This Paper Address?

Diffusion Transformers (DiTs) have become the state-of-the-art for high-fidelity image and video generation (powering models like Flux and Sora). However, their iterative denoising process is notoriously slow and computationally heavy.

Existing "training-free" acceleration methods usually rely on static caching—reusing features from previous timesteps at fixed intervals or specific layers. The authors argue this is inefficient because the denoising process isn't uniform: early stages focus on global structure, while later stages refine fine-grained textures. A "one-size-fits-all" caching strategy often leads to structural distortions or loss of detail. SortBlock addresses this by making the caching process adaptive and block-aware.

Methods

The SortBlock framework introduces three key components to make feature reuse smarter:

  1. Similarity-Aware Ranking: Instead of skipping blocks at random or on a fixed schedule, SortBlock calculates the cosine similarity of the "residuals" (the change between input and output) for each block across adjacent timesteps. It ranks these blocks and only recomputes the ones that have changed the most (lowest similarity), while skipping the redundant ones.
  2. Linear Prediction Mechanism: Simply "copying" a feature from a previous step (direct caching) leads to accumulated errors. SortBlock uses a lightweight linear extrapolation to predict what the feature should look like at the current step based on its previous trajectory. This first-order approximation significantly improves accuracy with almost zero computational overhead.
  3. Dynamic Recomputation Ratio ($\rho$): The authors observed that redundancy is highest in the middle of the denoising process. They use a 5th-degree polynomial curve to adaptively adjust the ratio of recomputed blocks at each stage of denoising, ensuring high fidelity during the critical early and late stages.

Datasets

The authors evaluated SortBlock across several leading-edge DiT architectures:

  • Flux.1-dev: A high-resolution text-to-image model.
  • Wan2.1 & HunyuanVideo: State-of-the-art text-to-video models.
  • Evaluation Benchmarks: The team used the COCO dataset for image metrics (FID, CLIP score) and VBench for comprehensive video quality assessment.

Key Insights and Ideas

  • The "70% Rule": Analysis showed that approximately 70% of diffusion steps—primarily in the middle of the process—exhibit very low variation in attention outputs. This is the "gold mine" for computational savings.
  • Block Sensitivity: Shallow blocks (near the input) capture global layouts, while deeper blocks refine details. SortBlock’s ability to treat these blocks differently based on real-time similarity is what allows it to maintain quality where static methods fail.
  • Prediction over Caching: The transition from "reusing" to "predicting" features is crucial. By capturing the linear trend of the feature trajectory, SortBlock avoids the "ghosting" or misalignment artifacts common in simpler caching schemes.

Key Results

SortBlock consistently outperformed existing methods like PAB, T-GATE, and TeaCache in the speed-vs-quality trade-off:

  • Flux.1-dev: Achieved a 2.39× speedup, reducing latency from 23.23s to 9.7s per image, while maintaining a higher Image Reward and SSIM than competitors.
  • Wan2.1 (Video): Delivered a 2.0× speedup (38s vs. 86s) with a VBench score of 79.47%, nearly identical to the 79.55% baseline.
  • HunyuanVideo: Achieved a 2.4× speedup while actually improving PSNR and SSIM compared to other acceleration baselines.

Limitations and Future Work

  • Hyperparameter Sensitivity: The method relies on a policy refresh interval ($K$) and a scaling hyperparameter ($\beta$). While the paper provides empirical defaults, these might need tuning for entirely new architectures.
  • Polynomial Fitting: The recomputation ratio curve is currently fitted to observed L1 distances. Automating this fitting process for any arbitrary model without manual analysis would be a valuable next step.
  • Hardware Specifics: While the speedups are significant on NVIDIA A800/H100 GPUs, the overhead of similarity calculations on lower-end consumer hardware remains an area for further investigation.

Summary updated

Abstract

Diffusion Transformers (DiTs) have demonstrated remarkable generative capabilities, particularly benefiting from Transformer architectures that enhance visual and artistic fidelity. However, their inherently sequential denoising process results in high inference latency, limiting their deployment in real-time scenarios. Existing training-free acceleration approaches typically reuse intermediate features at fixed timesteps or layers, overlooking the evolving semantic focus across denoising stages and Transformer blocks.To address this, we propose Sortblock, a training-free inference acceleration framework that dynamically caches block-wise features based on their similarity across adjacent timesteps. By ranking the evolution of residuals, Sortblock adaptively determines a recomputation ratio, selectively skipping redundant computations while preserving generation quality. Furthermore, we incorporate a lightweight linear prediction mechanism to reduce accumulated errors in skipped blocks.Extensive experiments across various tasks and DiT architectures demonstrate that Sortblock achieves over 2$\times$ inference speedup with minimal degradation in output quality, offering an effective and generalizable solution for accelerating diffusion-based generative models.
BibTeX
@inproceedings{chen2026sortblock,
  author = {Hanqi Chen and Xu Zhang and Xiaoliu Guan and Lielin Jiang and Guanzhong Wang and Zeyu Chen and Yi Liu},
  title = {{Sortblock: Similarity-Aware Feature Reuse for Diffusion Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i4.37276},
}

Loading related content...

Figure