Q&C: When Quantization Meets Cache in Efficient Image Generation

Xin Ding, Xin Li, Haotong Qin, Zhibo ChenarXiv 2025 · DOI · arXiv

Introduction

Diffusion Transformers (DiTs) have established themselves as the state-of-the-art architecture for high-quality image synthesis. However, their superior performance comes at a significant computational cost; generating a single 512×512 image can exceed 20 seconds on high-end hardware. To address this, researchers typically employ two primary acceleration strategies: Post-Training Quantization (PTQ), which reduces numerical precision (e.g., from 16-bit to 8-bit), and Feature Caching, which reuses computations from previous denoising steps to skip redundant processing.

While effective in isolation, the simultaneous application of these techniques has historically led to a "catastrophic" degradation in image quality. This paper introduces Q&C, a novel framework designed to resolve the negative synergy between quantization and caching, enabling massive speedups without sacrificing visual fidelity.

The Synergy Challenge: Why Models Fail

The researchers identified two fundamental reasons why combining quantization and caching breaks Diffusion Transformers:

1. Calibration Collapse

PTQ requires a small "calibration" dataset to determine the optimal scaling factors for weights and activations. When caching is active, the features across different timesteps become highly redundant—often exceeding 60% similarity as the generation process nears completion. This redundancy causes standard calibration methods to fail, as the calibration samples lose the diversity needed to map the model's full dynamic range.

2. Amplified Exposure Bias

The combination of quantization noise and approximation errors from caching creates a "snowball effect." This interaction shifts the variance of the model's output distribution. Unlike standard models where errors might remain localized, this shift in variance causes the sampling process to drift further away from the intended distribution at every step, resulting in distorted or noisy final images.

The Q&C Framework

To overcome these hurdles, the Q&C framework introduces two training-free components that stabilize the generation process.

Temporal-Aware Parallel Clustering (TAP)

To solve Calibration Collapse, the authors developed TAP. Instead of random sampling, TAP employs an efficient clustering algorithm that evaluates both spatial similarity (visual features) and temporal dynamics (how features evolve over time). By selecting the most informative and distinct samples across the entire diffusion trajectory, TAP ensures that the quantization process is calibrated against a representative roadmap of the model's behavior.

Variance Compensation (VC)

To mitigate exposure bias, the authors introduced Variance Compensation (VC). While previous attempts to fix distribution shifts often required training auxiliary neural networks, VC is entirely training-free. It uses a small batch of intermediate samples to mathematically calculate the variance shift at each timestep and applies an adaptive correction factor. This keeps the sampling distribution aligned with the original full-precision model throughout the denoising process.

Experimental Results

The researchers evaluated Q&C on standard high-resolution benchmarks, including ImageNet (256×256 and 512×512) and LSUN (Bedroom and Church).

  • Efficiency: Q&C achieved a 12.7× relative speedup compared to the standard DDPM baseline.
  • Quality: On ImageNet 256×256 using W8A8 (8-bit weights and activations) quantization, Q&C reached an FID of 5.43. This significantly outperforms other state-of-the-art methods like PTQ4DM (FID 17.52) and Q-Diffusion (FID 14.61) under identical caching conditions.
  • Pareto Optimality: In comparative analysis, Q&C consistently occupies the optimal region of the speed-vs-quality curve, providing the best trade-off between generation latency and Fréchet Inception Distance (FID).

Conclusion

Q&C demonstrates that the perceived incompatibility between quantization and caching is not an inherent limitation of Diffusion Transformers, but rather a result of unmanaged variance and redundant calibration. By introducing temporal-aware clustering and training-free variance compensation, Q&C allows for extreme acceleration (over 12×) with negligible impact on image quality. Future work may extend these insights into the realm of video generation, where the computational demands of Transformers are even more pronounced.

Summary updated

Abstract

Quantization and cache mechanisms are typically applied individually for efficient Diffusion Transformers (DiTs), each demonstrating notable potential for acceleration. However, the promoting effect of combining the two mechanisms on efficient generation remains under-explored. Through empirical investigation, we find that the combination of quantization and cache mechanisms for DiT is not straightforward, and two key challenges lead to severe catastrophic performance degradation: (i) the sample efficacy of calibration datasets in post-training quantization (PTQ) is significantly eliminated by cache operation; (ii) the combination of the above mechanisms introduces more severe exposure bias within sampling distribution, resulting in amplified error accumulation in the image generation process. In this work, we take advantage of these two acceleration mechanisms and propose a hybrid acceleration method by tackling the above challenges, aiming to further improve the efficiency of DiTs while maintaining excellent generation capability. Concretely, a temporal-aware parallel clustering (TAP) is designed to dynamically improve the sample selection efficacy for the calibration within PTQ for different diffusion steps. A variance compensation (VC) strategy is derived to correct the sampling distribution. It mitigates exposure bias through an adaptive correction factor generation. Extensive experiments have shown that our method has accelerated DiTs by 12.7x while preserving competitive generation capability. The code will be available at https://github.com/xinding-sys/Quant-Cache.
BibTeX
@misc{ding2025q,
  author = {Xin Ding and Xin Li and Haotong Qin and Zhibo Chen},
  title = {{Q\&C: When Quantization Meets Cache in Efficient Image Generation}},
  year = {2025},
  eprint = {2503.02508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.02508},
}
Figure