Abstract
Vision generative models have recently made significant advancements along two primary paradigms: diffusion-style and language-style, both of which have demonstrated excellent scaling laws. Quantization is crucial for efficiently deploying these models, as it reduces memory and computation costs. In this work, we systematically investigate the impact of quantization on these two paradigms. Surprisingly, despite achieving comparable performance in full precision, language-style models consistently outperform diffusion-style models across various quantization settings. This observation suggests that language-style models have superior bit-level scaling laws, offering a better tradeoff between model quality and total bits. To dissect this phenomenon, we conduct extensive experiments and find that the primary reason is the discrete representation space of language-style models, which is more tolerant of information loss during quantization. Furthermore, our analysis indicates that improving the bit-level scaling law of quantized vision generative models is challenging, with model distillation identified as a highly effective approach. Specifically, we propose TopKLD to optimize the transfer of distilled knowledge by balancing "implicit knowledge" and "explicit knowledge" during the distillation process. This approach elevates the bit-level scaling laws by one level across both integer and floating-point quantization settings.
Introduction
As vision generative models scale to billions of parameters, their deployment becomes increasingly resource-intensive. Quantization—reducing the precision of model weights and activations (e.g., from 16-bit to 4-bit)—is the standard industry solution to mitigate memory and compute overhead. However, the field lacks a comprehensive understanding of bit-level scaling laws: the quantitative relationship between model performance (e.g., FID) and the total bit budget. This paper investigates whether the two dominant architectural paradigms—diffusion-style (like DiT) and language-style (like VAR)—scale identically under quantization pressure.
The "Discrete Advantage" in Scaling
A central finding of this study is that language-style models (VAR) consistently outperform diffusion models (DiT) under quantization, even when they exhibit identical performance at full precision. When plotted on an FID-vs-Bits graph, quantized VAR models shift significantly toward the "lower-left" (lower FID at lower bit budgets) compared to their DiT counterparts.
As illustrated in the figures, VAR models maintain high generative quality even at aggressive quantization levels like W4A8 (4-bit weights, 8-bit activations). In contrast, DiT models experience a much sharper degradation in performance at similar bitwidths.
Mechanisms of Robustness
Why do language-style models handle quantization noise so much better? The researchers identify two primary factors:
- Codebooks as Error Absorbers: In autoregressive models, continuous features are eventually mapped to a discrete codebook. This discretization process naturally "absorbs" minor quantization errors. If the noise introduced by quantization is not large enough to push a feature vector to a different codebook entry, the effective error at the output is zero.
- Activation Stability: Diffusion models require denoising over multiple timesteps, during which activation distributions can vary wildly. This variance makes them inherently harder to quantize. Language-style models exhibit much more stable activation distributions, providing a more robust foundation for bit reduction.
TopKLD: Elevating Scaling Laws
To push the boundaries of ultra-low bitwidths (e.g., 3-bit), the authors introduce TopKLD, a novel distillation method. Standard Knowledge Distillation (KD) often struggles to balance "mode-seeking" (focusing on high-probability tokens) and "mode-covering" (capturing the full distribution). TopKLD addresses this by splitting the probability distribution:
- Explicit Knowledge: Employs a mode-seeking approach for the Top-K categories to ensure the model captures the most likely tokens.
- Implicit Knowledge: Uses a mode-covering approach for the remaining long-tail categories to preserve "dark knowledge."
Applying TopKLD elevates bit-level scaling laws by approximately "one level." For instance, a 3-bit model trained with TopKLD can achieve performance comparable to a standard 4-bit model.
Key Insights and Future Work
The study confirms that for generative quality, "wider and skinnier" is often superior: a large 4-bit model frequently outperforms a smaller 8-bit model with the same memory footprint. While W8A8 is currently the optimal configuration for language-style models, TopKLD allows W4A8 models to significantly close the quality gap.
Future research will need to address the limitations of continuous-token sequence models (like MAR), which currently lack the "discrete advantage" of codebook-based architectures. Additionally, while TopKLD is highly effective, its reliance on a full-precision teacher model and Quantization-Aware Training (QAT) presents a computational challenge for models exceeding 7B parameters.
Summary updated