Abstract
Diffusion models have achieved remarkable success in image and video generation tasks but require significant memory and time overhead during inference. Post-training quantization (PTQ) is a promising solution, yet existing methods struggle with large activation distribution variations across channels and timesteps, and input inconsistencies between calibration and inference. We propose Timestep-Channel Adaptive Quantization for Diffusion Models (TCAQ-DM), featuring a timestep-channel joint reparameterization (TCR) module to balance activation ranges, a dynamically adaptive quantization (DAQ) module to select optimal quantizers based on distribution types, and a progressively aligned reconstruction (PAR) strategy to mitigate input mismatch. TCAQ-DM significantly outperforms state-of-the-art methods, particularly in ultra-low bit settings like W4A4.
Introduction
Diffusion models generate high-quality content through an iterative denoising process. However, this process is computationally expensive. While PTQ can reduce inference costs, diffusion models present unique challenges:
- Temporal Volatility: Activation ranges vary significantly across denoising timesteps and channels.
- Distribution Shifts: Post-Softmax layers exhibit shifting distributions (e.g., from normal to power-law) as denoising progresses.
- Inference Mismatch: Standard PTQ optimizes blocks in isolation, ignoring the cumulative error in the iterative inference loop where errors in step $t$ affect the input for step $t-1$.
Methodology
TCAQ-DM addresses these challenges through three integrated components designed to handle the dynamic nature of diffusion activations.
Timestep-Channel Joint Reparameterization (TCR)
TCR balances activation ranges by calculating a scaling vector that aggregates values across all timesteps. This prevents outlier channels from dominating the quantization error while maintaining weight stability through reparameterization.
Dynamically Adaptive Quantizer (DAQ)
DAQ employs Maximum Likelihood Estimation (MLE) to identify the distribution type of post-Softmax layers. It adaptively selects between a log2 quantizer for power-law-like (long-tailed) distributions and a uniform quantizer for normal distributions, minimizing quantization error dynamically as the image clears up.
Progressively Aligned Reconstruction (PAR)
PAR addresses the "open-loop" problem of traditional PTQ. Instead of using a static calibration set, PAR iteratively samples new data from the currently quantized model during the reconstruction phase. This ensures the model is optimized for the actual data distribution it will encounter during the multi-step denoising process.
Experiments and Results
TCAQ-DM was evaluated on standard generative benchmarks including CIFAR-10, LSUN-Bedrooms, LSUN-Churches, and ImageNet.
Quantitative Performance
The quantitative improvements are most dramatic in low-bit settings. On CIFAR-10 (W4A4), TCAQ-DM achieved an FID of 6.38, whereas the previous state-of-the-art (TFMQ-DM) essentially failed with an FID of 236.63. At W6A6, TCAQ-DM (FID 4.40) nearly matches the full-precision (FP32) model's score of 4.14.
Qualitative Comparison
Visual results highlight the robustness of TCAQ-DM. While competing methods often collapse into unrecognizable noise at W4A4 bit-widths, TCAQ-DM maintains structural integrity and visual detail.
Conclusion
TCAQ-DM effectively handles the temporal and distributional complexities of diffusion models. By introducing timestep-aware reparameterization, adaptive quantization, and aligned reconstruction, it enables high-quality image generation even at ultra-low bit-widths, significantly reducing the resource requirements for deploying these models on edge devices.
Summary updated