TR-DQ: Time-Rotation Diffusion Quantization

Yihua Shao, Deyang Lin, Minxi Yan, Siyu Chen, Fanhu Zeng, Minwen Liao, Ao Ma, Ziyang Yan, Haozhe Wang, Yan Wang, Zhi Chen, Xiaofeng Cao, Haotong Qin, Hao Tang, Jingcai GuoAAAI 2026 · OpenReview

TL;DR

Diffusion models are powerful for generating images and videos but are computationally expensive. This paper introduces Time-Rotation Diffusion Quantization (TR-DQ), a novel method that significantly speeds up inference and reduces memory usage by dynamically quantizing activations and weights across different time steps. TR-DQ achieves state-of-the-art performance while offering substantial efficiency gains, making diffusion models more practical for deployment.

What Problem Does This Paper Address?

Diffusion models have revolutionized generative AI, producing incredibly realistic images and videos. However, their complex architectures and iterative sampling processes lead to high computational costs and memory demands, hindering their deployment on resource-constrained devices like edge hardware. Existing model compression techniques, particularly quantization (reducing the precision of model weights and activations), often struggle with diffusion models. This is because they typically overlook the crucial role of time-step variations during the sampling process and fail to adequately handle "massive outliers" in activations, which can lead to significant performance degradation. The paper aims to bridge this gap by developing a quantization method that specifically addresses these challenges in diffusion models.

Methods

The core of TR-DQ lies in its novel approach to quantization, focusing on two key areas:

  • Time-Rotation Diffusion Quantization (TR-DQ): This method dynamically quantizes activations and weights by incorporating time-step information.
  • Activation Smoothing via Rotation: TR-DQ uses a rotation matrix to transform activations, effectively smoothing out "massive outliers" and making their distribution more uniform. This is crucial because these outliers are difficult to quantize without significant loss.
  • Time-Dependent Quantization: Instead of using static quantization parameters for all time steps, TR-DQ introduces a time-dependent rotation matrix. This allows the quantization process to adapt to the changing activation distributions at each stage of the diffusion model's sampling process.
  • Balancing Strategy: The paper employs a multi-stage balancing strategy involving orthogonal rotation matrices and a "zigzag permutation" to ensure that both activations and weights are smoothed and their distributions are balanced across different blocks of the model. This helps to mitigate outliers more effectively.
  • Attention-Sharing Quantization (AS): For scenarios involving Classifier-Free Guidance (CFG), the authors observed significant similarities in attention values between conditional and unconditional paths. They propose an "Attention-Sharing" approach for certain blocks, merging these attention values to further reduce computational overhead and memory usage without compromising quality.

Datasets

The paper evaluates TR-DQ on standard image and video generation tasks.

  • Image Generation: PixArt-a (pre-trained on COCO dataset).
  • Video Generation: Open-SORA (pre-trained on UCF-101 dataset).

The paper does not introduce new benchmarks but rather leverages existing ones to demonstrate the effectiveness of their method.

Key Insights and Ideas

  • Time-Step Matters: The paper strongly emphasizes that ignoring the temporal dynamics of diffusion models during quantization is a major pitfall of previous methods. TR-DQ's success hinges on its ability to adapt quantization based on the current time step.
  • Outlier Handling is Key: The concept of "massive outliers" in activations is identified as a critical challenge. TR-DQ's rotation-based smoothing effectively addresses these outliers, which are difficult for standard quantization techniques to handle.
  • CFG Compression Potential: The observation that CFG and non-CFG paths share similar attention patterns opens up a new avenue for efficient compression, leading to the Attention-Sharing technique. This suggests that even complex features like CFG can be optimized.
  • Activations are the Bottleneck: The authors highlight that for diffusion models, the quantization of activations has a more significant impact on video generation quality than weight quantization, underscoring the importance of their activation-centric approach.

Key Results

TR-DQ demonstrates significant improvements in both generation quality and efficiency:

  • State-of-the-Art Performance: TR-DQ achieves state-of-the-art (SOTA) performance on image and video generation tasks across various metrics.
  • Efficiency Gains:
    • Speedup: 1.38-1.89× speedup in inference.
  • Memory Reduction: 1.97-2.58× memory reduction in inference.
  • Image Generation: TR-DQ outperforms other quantization methods, especially at lower bit-widths (e.g., 4-bit), maintaining high visual quality.
  • Video Generation: TR-DQ shows superior performance compared to SOTA methods like ViDiT-Q, particularly when quantizing to 4-bit weights. The paper suggests this is due to the model's better handling of temporal dependencies.
  • Attention-Sharing: The AS variant further enhances efficiency, showing a notable reduction in memory and latency (e.g., 2.17x memory and 1.91x latency reduction for 8/8 bit-width compared to the original model).

Limitations and Future Work

  • Limitations:
    • The paper notes that while Attention-Sharing (AS) improves efficiency, it can lead to a slight decrease in video generation quality, though the overall generative power remains strong.
  • The robustness of the VBench benchmark for video generation evaluation is mentioned as having potential errors.
  • The authors acknowledge that the AS technique might not be universally applicable to all diffusion models and requires careful analysis of attention similarities.
  • Future Work:
    • The paper suggests that the exploration of CFG-wise compression is a foundation for subsequent work, implying further research into optimizing CFG-enabled diffusion models.
  • The authors' findings on the significant impact of time-step modeling on video generation quality suggest that further research into time-aware diffusion model compression is warranted.
  • The paper's focus on dynamic quantization and outlier handling could be extended to other complex generative models.

Summary updated

Abstract

Diffusion models have been widely adopted in image and video generation. However, their complex network architecture leads to high inference overhead for its generation process. Existing diffusion quantization methods primarily focus on the quantization of the model structure while ignoring the impact of time-steps variation during sampling. At the same time, most current approaches fail to account for significant activations that cannot be eliminated, resulting in substantial performance degradation after quantization. To address these issues, we propose Time-Rotation Diffusion Quantization (TR-DQ), a novel quantization method incorporating time-step and rotation-based optimization. TR-DQ first divides the sampling process based on time-steps and applies a rotation matrix to smooth activations and weights dynamically. For different time-steps, a dedicated hyperparameter is introduced for adaptive timing modeling, which enables dynamic quantization across different time steps. Additionally, we also explore the compression potential of Classifier-Free Guidance (CFG-wise) to establish a foundation for subsequent work. TR-DQ achieves state-of-the-art (SOTA) performance on image generation and video generation tasks and a 1.38-1.89× speedup and 1.97-2.58× memory reduction in inference compared to existing quantization methods.
BibTeX
@inproceedings{shao2026tr,
  author = {Yihua Shao and Deyang Lin and Minxi Yan and Siyu Chen and Fanhu Zeng and Minwen Liao and Ao Ma and Ziyang Yan and Haozhe Wang and Yan Wang and Zhi Chen and Xiaofeng Cao and Haotong Qin and Hao Tang and Jingcai Guo},
  title = {{TR-DQ: Time-Rotation Diffusion Quantization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=wPpvvvCcck},
}
Figure