Looped Diffusion Transformers (DiTs) are a clever architectural trick to get the performance of massive models with a fraction of the parameters. By reusing the same transformer blocks multiple times in a "loop" during each denoising step, these models save on storage but still pay a high price in computation and memory during inference.
LoopPrune is a new framework designed specifically to compress these looped models. Unlike standard pruning methods that treat every layer as an independent entity, LoopPrune accounts for the fact that the same weights are being reused. By intelligently selecting which weights to cut and how to recover the lost information across all "loops," it achieves significant speedups and memory savings with remarkably little impact on image quality.
TL;DR
LoopPrune is the first pruning framework tailored for looped Diffusion Transformers, accounting for recurrent error propagation across shared blocks. It reduces GPU memory by up to 44% and accelerates denoising by up to 1.74x while maintaining significantly higher generation quality than standard pruning baselines.
What Problem Does This Paper Address?
Diffusion Transformers are the backbone of modern image generation, but they are notoriously resource-heavy. "Looped" variants (like SDXSv3 or Looped-DiT) mitigate this by reusing a small set of weights multiple times to simulate a deeper network.
However, pruning these models is tricky. If you use a "training-free" method (like Wanda or SparseGPT), you might ignore how a weight's importance changes from the first loop to the last. If you use a "training-based" method, you often focus on fixing local errors in one block, ignoring how those tiny errors compound and spiral out of control by the time the model reaches its final prediction. The authors identify a "recurrent gap": pruning errors in early loops propagate through the shared weights, leading to massive shifts in the final output.
Methods
LoopPrune tackles the "recurrent gap" through a three-stage pipeline:
- Recurrent-Aware Pruning: Instead of scoring a weight's importance based on a single pass, LoopPrune aggregates importance across all recurrent positions. It uses a clever scoring metric that combines the "shared effect" (average impact) and "variation" (to ensure it doesn't prune weights that are vital in some loops but not others).
- Training-Free Shared Reconstruction: Once a mask is chosen, the remaining weights are adjusted to compensate for the missing ones. Crucially, it pools input statistics from every loop iteration to find a "middle ground" weight update that works for the shared block's different roles.
- Gradient-Based Recurrent Recovery: This is the "fine-tuning" phase. Instead of just matching the output of a single block, LoopPrune matches the complete denoising prediction of the student model against a teacher model. It also explicitly matches the "text guidance" (the difference between conditional and unconditional prompts), which is highly sensitive to pruning noise.
Datasets
The authors evaluated LoopPrune across four open-source models (Looped-DiT B/16 & B/32, SDXSv3, and Surjo-Image-Preview) using:
- Calibration/Recovery: 256 COCO captions.
- Evaluation: A suite of six benchmarks covering text alignment (GenEval, DPG-Bench, TIIF-Bench), reasoning (T2I-CoReBench), and spatial understanding (PRISM-Bench, SpatialGenEval).
Key Insights and Ideas
- The Importance of "All Loops": The authors found that a channel's importance can vary wildly between the first and last loop. Selecting channels based on all loops results in much better consistency (42.96% vs 29.83% for first-loop only).
- Local Errors vs. Propagated Errors: A small error in a single transformer block might look negligible locally, but because that block is reused, the error compounds. Figure 5 shows that while local "module output" deviations might stay low, the "final prediction" deviations actually rise as the loops progress.
- Guidance is Fragile: The "text guidance" signal in diffusion models is 12x–22x more sensitive to pruning errors than the raw predictions themselves. LoopPrune’s explicit guidance matching is key to keeping the model "on track" with the user's prompt.
Key Results
LoopPrune sets a new state-of-the-art for diffusion pruning:
- Accuracy Retention: At 30% structured pruning, Looped-DiT B/16 retained 93.14% of its original accuracy, outperforming the best baseline by a massive 9.49%.
- Efficiency Gains: On the SDXSv3 model, LoopPrune achieved:
- 34.36% – 44.64% reduction in peak GPU memory.
- 1.46x – 1.74x speedup in denoising latency across PyTorch, vLLM, and SGLang backends.
- Versatility: It works across structured (removing whole heads/channels), unstructured (individual weights), and 2:4 semi-structured pruning paradigms.
Limitations and Future Work
- Recovery Overhead: While the pruning and reconstruction are fast, the "Recurrent Recovery" phase requires gradient-based updates. Although the authors use a very small dataset (256 captions), it's still more computationally expensive than purely training-free methods.
- Architecture Specificity: The method is specifically designed for looped architectures. While these are becoming more popular for edge deployment, the benefits might be less pronounced for standard "flat" DiTs.
- Future Directions: The authors suggest that combining LoopPrune with quantization or exploring even more aggressive "loop-skipping" strategies could further push the boundaries of diffusion efficiency.
Summary updated