TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models

Haocheng Huang, Jiaxin Chen, Jinyang Guo, Ruiyi Zhan, Yunhong Wang

TL;DR

TCAQ-DM is a post-training quantization (PTQ) framework that enables diffusion models to run efficiently at ultra-low bit-widths (down to 4-bit weights and activations). By dynamically adapting to severe activation shifts across channels and timesteps and realigning calibration data with iterative inference, it prevents the catastrophic generation failures that plague existing low-bit quantization methods.

What Problem Does This Paper Address?

Diffusion models generate stunning images and videos, but their inference cost is enormous due to large network backbones and the need for dozens or hundreds of sequential denoising steps. Post-training quantization (PTQ) compresses weights and activations into low-bit integers without expensive full retraining.

However, existing PTQ techniques break down when pushed to low bit-widths (like W6A6 or W4A4), causing severe image degradation or complete generative collapse. The authors identify three root causes:

  1. Severe Spatiotemporal Variance: Convolution activation ranges fluctuate wildly not just across channels, but across diffusion timesteps.
  2. Evolving Post-Softmax Distributions: Attention activations shift dynamically over the denoising trajectory, developing heavy "power-law" tails in later steps where static uniform quantizers fail.
  3. Reconstruction-Inference Mismatch: Typical PTQ methods calibrate layer-by-layer using isolated forward passes with clean full-precision inputs, ignoring how quantization errors compound over iterative multi-step sampling during real inference.

Methods

TCAQ-DM tackles these challenges across both the parameter initialization and block reconstruction stages:

  • Timestep-Channel Joint Reparameterization (TCR): To calm activation spikes in convolutional layers without multiplying stored weights, TCR calculates channel-wise scaling vectors across timesteps, weighted by activation magnitude. It mathematically shifts the dynamic range from activations into the static weights via linear reparameterization, bounding activation outliers.
  • Dynamically Adaptive Quantizer (DAQ): Rather than forcing attention post-Softmax layers into either a standard uniform or a logarithmic ($\log_2$) quantizer, DAQ fits activation profiles offline using Maximum Likelihood Estimation (MLE). If an activation distribution at a specific timestep strongly exhibits a power-law shape, it dynamically assigns a $\log_2$ quantizer; otherwise, it retains an efficient uniform quantizer.
  • Progressively Aligned Reconstruction (PAR): To address sampling mismatch, PAR alternates between block reconstruction and re-generating calibration data using the partially quantized model. This ensures intermediate calibration inputs reflect the accumulated errors encountered during real inference.
Initialization: 
  [Conv Layers]      ──> Timestep-Channel Reparameterization (TCR)
  [Attention/Softmax] ──> Distribution Test (MLE) ──> Dynamically Adaptive Quantizer (DAQ)

Reconstruction:
  [Quantized Blocks] ──> Iterative Sampling Loop ──> Progressively Aligned Reconstruction (PAR)

Datasets

The authors evaluated TCAQ-DM on standard generative benchmarks without introducing new datasets:

  • CIFAR-10 (unconditional generation with DDIM, 100 timesteps)
  • LSUN-Bedrooms and LSUN-Churches (unconditional generation with Latent Diffusion Model / LDM-4, 200 and 500 timesteps)
  • ImageNet (class-conditional generation with LDM-4, 20 timesteps)

Evaluation metrics include Fréchet Inception Distance (FID), Sliced FID (sFID), and Inception Score (IS).

Key Insights and Ideas

  • One scale vector across timesteps avoids weight explosion: Rescaling activations into weights is standard in Vision Transformers, but diffusion models reuse the same weights across hundreds of timesteps. Storing separate weights per step is unfeasible; TCR solves this by computing a global, activation-weighted channel scale factor that tames outliers across all timesteps simultaneously.
  • Softmax distributions are not static: While post-Softmax attention maps often exhibit power-law distributions, they only do so in certain stages of denoising. Forcing a $\log_2$ quantizer in early timesteps where activations occupy narrow, normal-like bands actually hurts accuracy. Dynamic per-step assignment prevents this error.
  • Quantization calibration must mimic multi-step inference: Reconstructing quantized blocks with pristine inputs creates an "optimism bias." Progressively feeding quantized intermediate outputs back into calibration exposes the optimization stage to true inference drift.

Key Results

TCAQ-DM sets a new state-of-the-art across low-bit diffusion quantization benchmarks, most notably salvaging regimes where prior methods fail completely:

  • Breakthrough on W4A4 (4-bit weights, 4-bit activations):
  • CIFAR-10 (DDIM): Achieves an FID of 6.38 (FP baseline is 4.14). Competing PTQ methods like Q-Diffusion (FID 384.21), PTQ4DM (FID 375.12), and TFMQ-DM (FID 236.63) completely collapse.
  • LSUN-Bedrooms (LDM-4): Achieves an FID of 16.43 (vs. 118.70 for TFMQ-DM and 334.83 for Q-Diffusion).
  • ImageNet (LDM-4): Achieves an FID of 30.69 (vs. 210.06 for TFMQ-DM and 376.54 for Q-Diffusion).
  • Near-Lossless W6A6: On CIFAR-10, TCAQ-DM achieves an FID of 4.40, nearly matching full-precision performance (4.14) and outperforming TFMQ-DM (9.59) and Q-Diffusion (26.06).
  • Efficiency: The offline MLE evaluation in DAQ introduces negligible computational overhead (~3% of overall quantization search time).

Limitations and Future Work

  • Remaining Gap in Extreme Quantization: While TCAQ-DM prevents generative collapse in 4-bit settings, a visible quality gap remains between W4A4 and full precision on complex datasets (e.g., ImageNet FID 30.69 vs. 10.91).
  • Offline Search Overhead: Progressively re-sampling calibration data across multiple rounds in PAR increases the overall time needed to quantize the model (even though inference latency is unaffected).
  • Scope: The method focuses on CNN and U-Net-based latent diffusion models; exploring these joint timestep-channel adaptations on pure Diffusion Transformers (DiTs) remains an open direction.

Summary updated

Abstract

Diffusion models have achieved remarkable success in the image and video generation tasks. Nevertheless, they often require a large amount of memory and time overhead during inference, due to the complex network architecture and considerable number of timesteps for iterative diffusion. Recently, the post-training quantization (PTQ) technique has proved a promising way to reduce the inference cost by quantizing the float-point operations to low-bit ones. However, most of them fail to tackle with the large variations in the distribution of activations across distinct channels and timesteps, as well as the inconsistent of input between quantization and inference on diffusion models, thus leaving much room for improvement. To address the above issues, we propose a novel method dubbed Timestep-Channel Adaptive Quantization for Diffusion Models (TCAQ-DM). Specifically, we develop a timestep-channel joint reparameterization (TCR) module to balance the activation range along both the timesteps and channels, facilitating the successive reconstruction procedure. Subsequently, we employ a dynamically adaptive quantization (DAQ) module that mitigate the quantization error by selecting an optimal quantizer for each post-Softmax layers according to their specific types of distributions. Moreover, we present a progressively aligned reconstruction (PAR) strategy to mitigate the bias caused by the input mismatch. Extensive experiments on various benchmarks and distinct diffusion models demonstrate that the proposed method substantially outperforms the state-of-the-art approaches in most cases, especially yielding comparable FID metrics to the full precision model on CIFAR-10 in the W6A6 setting, while enabling generating available images in the W4A4 settings.
BibTeX
@inproceedings{huang2025tcaq,
  author = {Haocheng Huang and Jiaxin Chen and Jinyang Guo and Ruiyi Zhan and Yunhong Wang},
  title = {{TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=FndjrF9SHv},
}

Loading related content...

Figure