TL;DR
TCAQ-DM is a post-training quantization (PTQ) framework that enables diffusion models to run efficiently at ultra-low bit-widths (down to 4-bit weights and activations). By dynamically adapting to severe activation shifts across channels and timesteps and realigning calibration data with iterative inference, it prevents the catastrophic generation failures that plague existing low-bit quantization methods.
What Problem Does This Paper Address?
Diffusion models generate stunning images and videos, but their inference cost is enormous due to large network backbones and the need for dozens or hundreds of sequential denoising steps. Post-training quantization (PTQ) compresses weights and activations into low-bit integers without expensive full retraining.
However, existing PTQ techniques break down when pushed to low bit-widths (like W6A6 or W4A4), causing severe image degradation or complete generative collapse. The authors identify three root causes:
- Severe Spatiotemporal Variance: Convolution activation ranges fluctuate wildly not just across channels, but across diffusion timesteps.
- Evolving Post-Softmax Distributions: Attention activations shift dynamically over the denoising trajectory, developing heavy "power-law" tails in later steps where static uniform quantizers fail.
- Reconstruction-Inference Mismatch: Typical PTQ methods calibrate layer-by-layer using isolated forward passes with clean full-precision inputs, ignoring how quantization errors compound over iterative multi-step sampling during real inference.
Methods
TCAQ-DM tackles these challenges across both the parameter initialization and block reconstruction stages:
- Timestep-Channel Joint Reparameterization (TCR): To calm activation spikes in convolutional layers without multiplying stored weights, TCR calculates channel-wise scaling vectors across timesteps, weighted by activation magnitude. It mathematically shifts the dynamic range from activations into the static weights via linear reparameterization, bounding activation outliers.
- Dynamically Adaptive Quantizer (DAQ): Rather than forcing attention post-Softmax layers into either a standard uniform or a logarithmic ($\log_2$) quantizer, DAQ fits activation profiles offline using Maximum Likelihood Estimation (MLE). If an activation distribution at a specific timestep strongly exhibits a power-law shape, it dynamically assigns a $\log_2$ quantizer; otherwise, it retains an efficient uniform quantizer.
- Progressively Aligned Reconstruction (PAR): To address sampling mismatch, PAR alternates between block reconstruction and re-generating calibration data using the partially quantized model. This ensures intermediate calibration inputs reflect the accumulated errors encountered during real inference.
Initialization:
[Conv Layers] ──> Timestep-Channel Reparameterization (TCR)
[Attention/Softmax] ──> Distribution Test (MLE) ──> Dynamically Adaptive Quantizer (DAQ)
Reconstruction:
[Quantized Blocks] ──> Iterative Sampling Loop ──> Progressively Aligned Reconstruction (PAR)
Datasets
The authors evaluated TCAQ-DM on standard generative benchmarks without introducing new datasets:
- CIFAR-10 (unconditional generation with DDIM, 100 timesteps)
- LSUN-Bedrooms and LSUN-Churches (unconditional generation with Latent Diffusion Model / LDM-4, 200 and 500 timesteps)
- ImageNet (class-conditional generation with LDM-4, 20 timesteps)
Evaluation metrics include Fréchet Inception Distance (FID), Sliced FID (sFID), and Inception Score (IS).
Key Insights and Ideas
- One scale vector across timesteps avoids weight explosion: Rescaling activations into weights is standard in Vision Transformers, but diffusion models reuse the same weights across hundreds of timesteps. Storing separate weights per step is unfeasible; TCR solves this by computing a global, activation-weighted channel scale factor that tames outliers across all timesteps simultaneously.
- Softmax distributions are not static: While post-Softmax attention maps often exhibit power-law distributions, they only do so in certain stages of denoising. Forcing a $\log_2$ quantizer in early timesteps where activations occupy narrow, normal-like bands actually hurts accuracy. Dynamic per-step assignment prevents this error.
- Quantization calibration must mimic multi-step inference: Reconstructing quantized blocks with pristine inputs creates an "optimism bias." Progressively feeding quantized intermediate outputs back into calibration exposes the optimization stage to true inference drift.
Key Results
TCAQ-DM sets a new state-of-the-art across low-bit diffusion quantization benchmarks, most notably salvaging regimes where prior methods fail completely:
- Breakthrough on W4A4 (4-bit weights, 4-bit activations):
- CIFAR-10 (DDIM): Achieves an FID of 6.38 (FP baseline is 4.14). Competing PTQ methods like Q-Diffusion (FID 384.21), PTQ4DM (FID 375.12), and TFMQ-DM (FID 236.63) completely collapse.
- LSUN-Bedrooms (LDM-4): Achieves an FID of 16.43 (vs. 118.70 for TFMQ-DM and 334.83 for Q-Diffusion).
- ImageNet (LDM-4): Achieves an FID of 30.69 (vs. 210.06 for TFMQ-DM and 376.54 for Q-Diffusion).
- Near-Lossless W6A6: On CIFAR-10, TCAQ-DM achieves an FID of 4.40, nearly matching full-precision performance (4.14) and outperforming TFMQ-DM (9.59) and Q-Diffusion (26.06).
- Efficiency: The offline MLE evaluation in DAQ introduces negligible computational overhead (~3% of overall quantization search time).
Limitations and Future Work
- Remaining Gap in Extreme Quantization: While TCAQ-DM prevents generative collapse in 4-bit settings, a visible quality gap remains between W4A4 and full precision on complex datasets (e.g., ImageNet FID 30.69 vs. 10.91).
- Offline Search Overhead: Progressively re-sampling calibration data across multiple rounds in PAR increases the overall time needed to quantize the model (even though inference latency is unaffected).
- Scope: The method focuses on CNN and U-Net-based latent diffusion models; exploring these joint timestep-channel adaptations on pure Diffusion Transformers (DiTs) remains an open direction.
Summary updated