TL;DR
APQ-DM is a novel post-training quantization (PTQ) framework designed to make diffusion models significantly more efficient without sacrificing image quality. By employing distribution-aware quantization groups and an active timestep selection strategy for calibration, it achieves state-of-the-art performance, enabling high-quality 6-bit and 8-bit image generation with negligible computational overhead.
What Problem Does This Paper Address?
Denoising diffusion models have revolutionized generative AI, but their iterative nature makes them computationally expensive and slow to deploy on edge devices like phones or robots. Quantization—reducing the precision of weights and activations (e.g., from 32-bit floats to 8-bit integers)—is a standard solution for acceleration.
However, standard PTQ methods struggle with diffusion models for two main reasons:
- Non-stationary Distributions: Unlike standard classifiers, a diffusion model's activation distributions shift dramatically across hundreds of denoising timesteps, moving from pure Gaussian noise to structured image features. Using a single, shared quantization function for all steps introduces massive discretization errors.
- Suboptimal Calibration: PTQ requires a small "calibration" dataset to tune quantization parameters. Conventional methods pick random timesteps to generate these calibration images, which often fails to capture the most informative parts of the generation process, leading to poor generalization.
Methods
The authors introduce APQ-DM (Accurate Post-training Quantization for Diffusion Models) to tackle these issues through two core innovations:
1. Distribution-aware Quantization
Instead of a one-size-fits-all approach, APQ-DM partitions the generation timesteps into groups. Each group learns its own specific rounding and clipping functions tailored to the activation distributions of those steps. To find the best grouping without an exhaustive search, the authors use a differentiable search framework. During this search, activations are processed by multiple quantization functions and combined using learnable "importance weights." After optimization, the function with the highest weight is selected for each timestep.
2. Active Calibration Timestep Selection
To build a better calibration set, the authors move away from random sampling. They extend the Structural Risk Minimization (SRM) principle to actively select the most "informative" timesteps. The selection criteria balance two factors:
- Uncertainty (Exploitation): Selecting timesteps where the quantization importance weights have high entropy (meaning the model is "unsure" which group is best).
- Diversity (Exploration): Selecting timesteps that haven't been sampled frequently yet.
Datasets
The framework was evaluated across a wide range of standard generative benchmarks:
- Unconditional Generation: CIFAR-10 (32x32), CelebA (64x64), LSUN-Bedroom (256x256), and LSUN-Church (256x256).
- High-Resolution & Latent Diffusion: CelebA-HQ (256x256) and ImageNet (256x256) using Latent Diffusion Models (LDMs).
- Text-to-Image: Evaluated qualitatively using Stable Diffusion.
Key Insights and Ideas
- Grouping is the "Goldilocks" Zone: The authors found that using 8 groups for timestep partitioning provides the best trade-off. Too few groups (like 1) lead to high quantization error, while too many (like per-step quantization) lead to overfitting due to the limited calibration data.
- Informative Samples Matter More Than Quantity: Active sampling allows the model to achieve better results with a calibration set of only 256 images than random sampling achieves with 1024 images.
- Sensitivity to Clipping: The differentiable search is particularly sensitive to the hyperparameter $\lambda$, which controls the discretization loss. This highlights that managing the "clipping vs. rounding" error trade-off is critical when activation distributions are wide.
Key Results
APQ-DM consistently outperforms existing state-of-the-art PTQ methods like PTQ4DM and Q-Diffusion across all bitwidths and datasets.
- Quantitative Gains: On the LSUN-Bedroom dataset (W6A6 quantization), APQ-DM achieved an FID of 15.72, significantly better than PTQ4DM's 16.85. On ImageNet (W8A8), it improved the Inception Score (IS) from 161.75 (PTQ4DM) to 179.13.
- Qualitative Quality: Even at 6-bit precision, APQ-DM generates images that are visually nearly indistinguishable from full-precision (32-bit) models, maintaining sharp details that other quantization methods lose.
Limitations and Future Work
- Search Overhead: While inference is highly efficient, the framework requires an offline differentiable search phase to determine group assignments and quantization parameters.
- Ultra-low Bitwidths: The paper focuses on 6-bit and 8-bit quantization. Pushing diffusion models to 4-bit or lower remains a significant challenge that likely requires more aggressive quantization-aware training (QAT) rather than just post-training methods.
- Future Direction: The authors suggest that this distribution-aware approach could be extended to other non-stationary tasks, such as video generation or reinforcement learning agents.
Summary updated