Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models

Xin Ding, Shijie Cao, Ting Cao, Zhibo ChenSubmission to ICLR 2025 · Rejected

TL;DR

While large language models benefit heavily from low-bit quantization scaling laws (where a larger, lower-bit model often outperforms a smaller, higher-bit model at the same bit budget), visual generative models have remained under-explored. This paper investigates bit-level scaling laws across visual generative paradigms, discovering that discrete language-style models (like VAR and LlamaGen) tolerate quantization far better than continuous diffusion models (like DiT). To push low-bit scaling even further, the authors propose TopKLD, a hybrid distillation method that balances mode-seeking and mode-covering knowledge transfer, revitalizing 3-bit and 4-bit performance.

What Problem Does This Paper Address?

As visual generative models scale into billions of parameters, deploying them requires aggressive quantization to curb memory footprint and latency. In NLP, bit-level scaling laws predict that allocating bits to model size rather than numerical precision (e.g., running a 4-bit 6B model instead of an 8-bit 3B model) yields superior performance for the same total bit budget.

However, visual generation is dominated by two divergent paradigms: continuous diffusion-style models (e.g., DiT) and discrete language-style autoregressive models (e.g., VAR, LlamaGen). Do both paradigms scale equally well under quantization? The authors set out to determine whether diffusion and autoregressive vision models exhibit similar bit-level scaling behaviors and how to optimize vision architectures for low-bit deployment.

Methods

  1. Systematic Bit-Level Scaling Benchmarking: The authors evaluate models ranging from 300M to 7B parameters across precisions from 16-bit down to 3-bit (W3A16, W4A16, W8A16, W4A8, W8A8) across both post-training quantization (PTQ) and quantization-aware training (QAT).
  2. Error & Activation Analysis: They isolate errors in two generative phases: feature extraction and representation space reconstruction, measuring single-step error sensitivity and multi-step error accumulation using controlled Gaussian noise injections.
  3. TopKLD Distillation: Standard distillation uses either Forward KL (mode-covering, which spreads probability mass) or Reverse KL (mode-seeking, focusing on dominant modes but ignoring subtle "implicit" non-target logits). To balance these, TopKLD partitions the output vocabulary into top-$K$ categories ($M_s$) and the remaining tail categories ($M_c$): $$\text{TopKLD}(P_T \parallel P_S) = \sum_{t=1}^T \left[ \sum_{y' \in M_s} P_S \log \frac{P_S}{P_T} + \sum_{y' \in M_c} P_T \log \frac{P_T}{P_S} \right]$$ It applies Reverse KL to the high-probability tokens (capturing explicit mode choices) and Forward KL to the tail tokens (preserving implicit contextual cues).

Datasets

  • ImageNet $256 \times 256$: Standard benchmark for class-conditional image synthesis.
  • Evaluated Architectures:
  • Diffusion-style (continuous): DiT (458M to 7B parameters).
  • Language-style (discrete): VAR (310M to 2B parameters) and LlamaGen (343M to 3.1B parameters).
  • Language-style (continuous): MAR (208M to 943M parameters), used to isolate the effect of continuous vs. discrete token spaces.
  • No new datasets were introduced.

Key Insights and Ideas

  • Discrete representation spaces absorb quantization error: Language-style models (VAR, LlamaGen) demonstrate far superior bit-level scaling over full precision, shifting scaling curves down and to the left (better FID at lower compute/memory budgets). Conversely, diffusion models (DiT) show almost no bit-level scaling advantage over full precision.
  • The codebook acts as a noise buffer: Mapping continuous features into discrete codebook indices rounds small quantization perturbations back to the correct centroid. In contrast, diffusion models operate in continuous latent spaces where quantization noise directly corrupts intermediate states and compounds over sampling steps.
  • Continuous language models still fail: Testing MAR (an autoregressive model operating in continuous space) confirmed that it fails to exhibit bit-level scaling benefits, proving that the discrete codebook—not merely the autoregressive sequence structure—is the root driver of quantization robustness.
  • Activation stability: Language-style models show far more consistent layer-wise activation variance across generation timesteps than DiT, drastically simplifying activation quantization.

Key Results

  • Superior Scaling of Quantized VAR: Quantized VAR at W4A16 and W8A8 consistently surpasses its full-precision W16A16 baseline when evaluated under identical total parameter memory and compute budgets.
  • Failure of Off-the-shelf PTQ at Extremely Low Bits: Advanced PTQ techniques (e.g., OmniQuant, GPTQ, GPTVQ) degrade severely at 3-bit weights (e.g., VAR-d30 W3A16 FID jumps to 13.48 with GPTQ and 3.93 with OmniQuant, compared to 1.92 in FP16).
  • TopKLD Restores and Boosts Scaling:
  • Under W3A16, TopKLD achieves an FID of 2.25 on VAR-d30 (2B parameters), outperforming standard Forward-KLD (2.55) and Reverse-KLD (2.55).
  • TopKLD enables 3-bit weight models (W3A16) to beat the scaling trajectory of standard 4-bit models. Under weight-activation quantization, W4A8 with TopKLD matches or exceeds W8A8 scaling limits.
  • Applying TopKLD to floating-point quantization formats yields further scaling enhancements over integer formats.

Limitations and Future Work

  • Scope of Architectures and Tasks: Evaluations focus primarily on class-conditional ImageNet generation with DiT, VAR, MAR, and LlamaGen; broader architectures (e.g., text-to-image diffusion pipelines like Stable Diffusion / Flux) and higher-resolution settings remain to be tested.
  • Continuous-Valued Sequence Models: Emerging visual tokenizers without codebooks (continuous-valued representations) struggle under this paradigm; future work is needed to adapt quantization strategies specifically for them.
  • Training Overhead: While PTQ is fast, the best-performing low-bit regimes (W3A16, W4A8) rely on quantization-aware fine-tuning with TopKLD distillation, introducing training computation costs.

Summary updated

Abstract

Vision generative models have recently made significant advancements along two primary paradigms: diffusion-style and language-style, both of which have demonstrated excellent scaling laws. Quantization is crucial for efficiently deploying these models, as it reduces memory and computation costs. In this work, we systematically investigate the impact of quantization on these two paradigms. Surprisingly, despite achieving comparable performance in full precision, language-style models consistently outperform diffusion-style models across various quantization settings. This observation suggests that language-style models have superior bit-level scaling laws, offering a better tradeoff between model quality and total bits. To dissect this phenomenon, we conduct extensive experiments and find that the primary reason is the discrete representation space of language-style models, which is more tolerant of information loss during quantization. Furthermore, our analysis indicates that improving the bit-level scaling law of quantized vision generative models is challenging, with model distillation identified as a highly effective approach. Specifically, we propose TopKLD to optimize the transfer of distilled knowledge by balancing "implicit knowledge" and "explicit knowledge" during the distillation process. This approach elevates the bit-level scaling laws by one level across both integer and floating-point quantization settings.
BibTeX
@misc{ding2025dissecting,
  author = {Xin Ding and Shijie Cao and Ting Cao and Zhibo Chen},
  title = {{Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models}},
  year = {2025},
  eprint = {2501.06218},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.06218},
}

Loading related content...

Figure