TL;DR
SortBlock is a training-free acceleration framework for Diffusion Transformers (DiTs) that achieves over 2× inference speedup with negligible loss in image and video quality. By dynamically identifying and skipping redundant computations in specific transformer blocks based on their similarity across timesteps, it bypasses the need for expensive retraining or distillation.
What Problem Does This Paper Address?
Diffusion Transformers (DiTs) have become the state-of-the-art for high-fidelity image and video generation (powering models like Flux and Sora). However, their iterative denoising process is notoriously slow and computationally heavy.
Existing "training-free" acceleration methods usually rely on static caching—reusing features from previous timesteps at fixed intervals or specific layers. The authors argue this is inefficient because the denoising process isn't uniform: early stages focus on global structure, while later stages refine fine-grained textures. A "one-size-fits-all" caching strategy often leads to structural distortions or loss of detail. SortBlock addresses this by making the caching process adaptive and block-aware.
Methods
The SortBlock framework introduces three key components to make feature reuse smarter:
- Similarity-Aware Ranking: Instead of skipping blocks at random or on a fixed schedule, SortBlock calculates the cosine similarity of the "residuals" (the change between input and output) for each block across adjacent timesteps. It ranks these blocks and only recomputes the ones that have changed the most (lowest similarity), while skipping the redundant ones.
- Linear Prediction Mechanism: Simply "copying" a feature from a previous step (direct caching) leads to accumulated errors. SortBlock uses a lightweight linear extrapolation to predict what the feature should look like at the current step based on its previous trajectory. This first-order approximation significantly improves accuracy with almost zero computational overhead.
- Dynamic Recomputation Ratio ($\rho$): The authors observed that redundancy is highest in the middle of the denoising process. They use a 5th-degree polynomial curve to adaptively adjust the ratio of recomputed blocks at each stage of denoising, ensuring high fidelity during the critical early and late stages.
Datasets
The authors evaluated SortBlock across several leading-edge DiT architectures:
- Flux.1-dev: A high-resolution text-to-image model.
- Wan2.1 & HunyuanVideo: State-of-the-art text-to-video models.
- Evaluation Benchmarks: The team used the COCO dataset for image metrics (FID, CLIP score) and VBench for comprehensive video quality assessment.
Key Insights and Ideas
- The "70% Rule": Analysis showed that approximately 70% of diffusion steps—primarily in the middle of the process—exhibit very low variation in attention outputs. This is the "gold mine" for computational savings.
- Block Sensitivity: Shallow blocks (near the input) capture global layouts, while deeper blocks refine details. SortBlock’s ability to treat these blocks differently based on real-time similarity is what allows it to maintain quality where static methods fail.
- Prediction over Caching: The transition from "reusing" to "predicting" features is crucial. By capturing the linear trend of the feature trajectory, SortBlock avoids the "ghosting" or misalignment artifacts common in simpler caching schemes.
Key Results
SortBlock consistently outperformed existing methods like PAB, T-GATE, and TeaCache in the speed-vs-quality trade-off:
- Flux.1-dev: Achieved a 2.39× speedup, reducing latency from 23.23s to 9.7s per image, while maintaining a higher Image Reward and SSIM than competitors.
- Wan2.1 (Video): Delivered a 2.0× speedup (38s vs. 86s) with a VBench score of 79.47%, nearly identical to the 79.55% baseline.
- HunyuanVideo: Achieved a 2.4× speedup while actually improving PSNR and SSIM compared to other acceleration baselines.
Limitations and Future Work
- Hyperparameter Sensitivity: The method relies on a policy refresh interval ($K$) and a scaling hyperparameter ($\beta$). While the paper provides empirical defaults, these might need tuning for entirely new architectures.
- Polynomial Fitting: The recomputation ratio curve is currently fitted to observed L1 distances. Automating this fitting process for any arbitrary model without manual analysis would be a valuable next step.
- Hardware Specifics: While the speedups are significant on NVIDIA A800/H100 GPUs, the overhead of similarity calculations on lower-end consumer hardware remains an area for further investigation.
Summary updated