TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models

Haocheng Huang, Jiaxin Chen, Jinyang Guo, Ruiyi Zhan, Yunhong Wang

Abstract

Diffusion models have achieved remarkable success in image and video generation tasks but require significant memory and time overhead during inference. Post-training quantization (PTQ) is a promising solution, yet existing methods struggle with large activation distribution variations across channels and timesteps, and input inconsistencies between calibration and inference. We propose Timestep-Channel Adaptive Quantization for Diffusion Models (TCAQ-DM), featuring a timestep-channel joint reparameterization (TCR) module to balance activation ranges, a dynamically adaptive quantization (DAQ) module to select optimal quantizers based on distribution types, and a progressively aligned reconstruction (PAR) strategy to mitigate input mismatch. TCAQ-DM significantly outperforms state-of-the-art methods, particularly in ultra-low bit settings like W4A4.

Introduction

Diffusion models generate high-quality content through an iterative denoising process. However, this process is computationally expensive. While PTQ can reduce inference costs, diffusion models present unique challenges:

  1. Temporal Volatility: Activation ranges vary significantly across denoising timesteps and channels.
  2. Distribution Shifts: Post-Softmax layers exhibit shifting distributions (e.g., from normal to power-law) as denoising progresses.
  3. Inference Mismatch: Standard PTQ optimizes blocks in isolation, ignoring the cumulative error in the iterative inference loop where errors in step $t$ affect the input for step $t-1$.

Methodology

TCAQ-DM addresses these challenges through three integrated components designed to handle the dynamic nature of diffusion activations.

The TCAQ-DM framework integrates Timestep-Channel Joint Reparameterization (TCR), Dynamically Adaptive Quantizer (DAQ), and Progressively Aligned Reconstruction (PAR).

Timestep-Channel Joint Reparameterization (TCR)

TCR balances activation ranges by calculating a scaling vector that aggregates values across all timesteps. This prevents outlier channels from dominating the quantization error while maintaining weight stability through reparameterization.

Dynamically Adaptive Quantizer (DAQ)

DAQ employs Maximum Likelihood Estimation (MLE) to identify the distribution type of post-Softmax layers. It adaptively selects between a log2 quantizer for power-law-like (long-tailed) distributions and a uniform quantizer for normal distributions, minimizing quantization error dynamically as the image clears up.

Progressively Aligned Reconstruction (PAR)

PAR addresses the "open-loop" problem of traditional PTQ. Instead of using a static calibration set, PAR iteratively samples new data from the currently quantized model during the reconstruction phase. This ensures the model is optimized for the actual data distribution it will encounter during the multi-step denoising process.

Experiments and Results

TCAQ-DM was evaluated on standard generative benchmarks including CIFAR-10, LSUN-Bedrooms, LSUN-Churches, and ImageNet.

Quantitative Performance

The quantitative improvements are most dramatic in low-bit settings. On CIFAR-10 (W4A4), TCAQ-DM achieved an FID of 6.38, whereas the previous state-of-the-art (TFMQ-DM) essentially failed with an FID of 236.63. At W6A6, TCAQ-DM (FID 4.40) nearly matches the full-precision (FP32) model's score of 4.14.

Qualitative Comparison

Visual results highlight the robustness of TCAQ-DM. While competing methods often collapse into unrecognizable noise at W4A4 bit-widths, TCAQ-DM maintains structural integrity and visual detail.

Visual comparison of generation results. TCAQ-DM avoids the collapse seen in previous methods at W4A4, producing recognizable images.

Conclusion

TCAQ-DM effectively handles the temporal and distributional complexities of diffusion models. By introducing timestep-aware reparameterization, adaptive quantization, and aligned reconstruction, it enables high-quality image generation even at ultra-low bit-widths, significantly reducing the resource requirements for deploying these models on edge devices.

Summary updated

Abstract

Diffusion models have achieved remarkable success in the image and video generation tasks. Nevertheless, they often require a large amount of memory and time overhead during inference, due to the complex network architecture and considerable number of timesteps for iterative diffusion. Recently, the post-training quantization (PTQ) technique has proved a promising way to reduce the inference cost by quantizing the float-point operations to low-bit ones. However, most of them fail to tackle with the large variations in the distribution of activations across distinct channels and timesteps, as well as the inconsistent of input between quantization and inference on diffusion models, thus leaving much room for improvement. To address the above issues, we propose a novel method dubbed Timestep-Channel Adaptive Quantization for Diffusion Models (TCAQ-DM). Specifically, we develop a timestep-channel joint reparameterization (TCR) module to balance the activation range along both the timesteps and channels, facilitating the successive reconstruction procedure. Subsequently, we employ a dynamically adaptive quantization (DAQ) module that mitigate the quantization error by selecting an optimal quantizer for each post-Softmax layers according to their specific types of distributions. Moreover, we present a progressively aligned reconstruction (PAR) strategy to mitigate the bias caused by the input mismatch. Extensive experiments on various benchmarks and distinct diffusion models demonstrate that the proposed method substantially outperforms the state-of-the-art approaches in most cases, especially yielding comparable FID metrics to the full precision model on CIFAR-10 in the W6A6 setting, while enabling generating available images in the W4A4 settings.
BibTeX
@inproceedings{huang2025tcaq,
  author = {Haocheng Huang and Jiaxin Chen and Jinyang Guo and Ruiyi Zhan and Yunhong Wang},
  title = {{TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=FndjrF9SHv},
}
Figure