Abstract
Diffusion models have demonstrated remarkable performance on vision generation tasks. However, the high computational complexity hinders its wide application on edge devices. Quantization has emerged as a promising technique for inference acceleration and memory reduction. However, existing quantization methods do not generalize well under extremely low-bit (2-4 bit) quantization. Directly applying these methods will cause severe performance degradation. We identify that the existing quantization framework suffers from the outlier-unfriendly quantizer design, suboptimal initialization, and optimization strategy. We present MPQ-DMv2, an improved Mixed Precision Quantization framework for extremely low-bit Diffusion Models. For the quantization perspective, the imbalanced distribution caused by salient outliers is quantization-unfriendly for uniform quantizer. We propose Flexible Z-Order Residual Mixed Quantization that utilizes an efficient binary residual branch for flexible quant steps to handle salient error. For the optimization framework, we theoretically analyzed the convergence and optimality of the LoRA module and propose Object-Oriented Low-Rank Initialization to use prior quantization error for informative initialization. We then propose Memory-based Temporal Relation Distillation to construct an online time-aware pixel queue for long-term denoising temporal information distillation, which ensures the overall temporal consistency between quantized and full-precision model. Comprehensive experiments on various generation tasks show that our MPQ-DMv2 surpasses current SOTA methods by a great margin on different architectures, especially under extremely low-bit widths.
Introduction
Diffusion Models (DMs) like Stable Diffusion have set new benchmarks in image generation but remain computationally expensive due to their iterative denoising process. Deploying these models on edge devices necessitates aggressive quantization (e.g., 2-4 bits). However, standard quantization techniques struggle with DMs for three main reasons:
- Salient Outliers: The weight distributions in DMs often contain extreme values that degrade the performance of uniform quantizers by stretching the quantization range.
- Temporal Drift: Denoising is a sequential process. Errors in early steps accumulate over time, leading to inconsistent trajectories compared to full-precision (FP) models.
- Suboptimal Initialization: Fine-tuning quantized models with Low-Rank Adaptation (LoRA) typically starts from a "cold" zero-initialization, ignoring the structured nature of the quantization error it aims to correct.
Methodology
MPQ-DMv2 addresses these challenges through three integrated components that optimize the quantization process from weight representation to temporal consistency.
Flexible Z-Order Residual Mixed Quantization (FZ-RMQ)
To handle outliers without sacrificing the precision of the majority of weights, FZ-RMQ employs a hierarchical structure. It uses a base $(n-1)$-bit quantizer for the bulk distribution and adds a lightweight binary residual branch. This dual-quantizer design effectively captures high-magnitude outliers, significantly reducing the overall quantization error compared to standard uniform quantizers.
Memory-based Temporal Relation Distillation (MTRD)
To maintain denoising consistency, MTRD utilizes an online Time-aware Pixel Queue. By caching features from various timesteps of the FP model, the quantized model is trained to match the FP model's temporal relationships using Kullback-Leibler (KL) divergence. This ensures the quantized model follows the correct denoising trajectory across the entire generation process.
Object-Oriented Low-Rank Initialization (OOLRI)
Instead of zero-initialization, OOLRI performs Singular Value Decomposition (SVD) on the initial quantization error matrix ($E = W - Q(W)$). These SVD components are used to initialize LoRA weights, providing a "warm start" that directly targets the most significant errors introduced by quantization, leading to faster and more stable convergence.
Experiments and Results
The framework was evaluated on LDM, Stable Diffusion v1.4, and DiT architectures across diverse datasets including LSUN-Bedrooms, LSUN-Churches, and MS-COCO.
Performance and Fidelity
MPQ-DMv2 significantly outperforms previous state-of-the-art (SOTA) methods. For instance, on the LSUN-Churches benchmark under the W2A4 setting, it achieved an FID of 15.20, a substantial improvement over the previous SOTA (MPQ-DM) at 21.83. It also showed remarkable robustness on the Diffusion Transformer (DiT-XL/2), improving FID from 130.34 to 35.64 in the W3A8 setting.
Efficiency Gains
The method achieves up to 15.8x model compression (e.g., LDM-4 shrinking from 1529MB to 96MB) and a 3.35x runtime speedup. Notably, the calibration process is highly efficient, requiring only approximately 3 hours of fine-tuning to recover performance.
Conclusion
MPQ-DMv2 provides a robust framework for extremely low-bit quantization of diffusion models. By addressing outliers via residual branches, ensuring temporal consistency through distillation, and utilizing error-informed initialization, it enables high-fidelity generation on resource-constrained devices. Future work includes extending this approach to video diffusion models where temporal consistency is even more critical.
Summary updated