TL;DR
TempQuant is a lightweight method to recover the accuracy lost when quantizing Large Language Models (LLMs) by automatically tuning decoding parameters like temperature and top-p. By matching the quantized model's probability distributions to its full-precision counterpart, it can significantly boost performance—for example, raising 3-bit Qwen3-4B’s MATH accuracy from 34% to 51%—without requiring any labeled data or model retraining.
What Problem Does This Paper Address?
Quantization (reducing the precision of model weights to 3 or 4 bits) is essential for running massive LLMs on consumer hardware, but it often degrades accuracy. Standard practice is to simply use the same decoding settings—specifically sampling temperature ($\tau$) and top-p ($\rho$)—as the original full-precision model.
The authors show this is a mistake. Quantization distorts the model's next-token probability distributions, effectively "warming" the model and increasing its output entropy (randomness). This can lead to repetitive loops or nonsensical outputs. The paper investigates whether adjusting these simple "knobs" at the decoding stage can undo the damage caused by quantization.
Methods
The core idea of TempQuant is to find new temperature and top-p settings that minimize the Kullback–Leibler (KL) divergence between the quantized model and the original full-precision (BF16) model. They propose two versions:
- Global TempQuant: This finds a single optimal $(\tau, \rho)$ pair for the entire model for a given requested temperature. It uses a simple grid search on a small set of unlabeled data. Because it only changes two numbers in the sampler, it adds zero cost to deployment.
- Per-step TempQuant: This uses a tiny MLP "head" (only ~165k parameters) to predict a custom temperature correction and top-p cutoff for every single token generated. It looks at features of the quantized logits (like entropy and gaps between top tokens) to decide how to adjust the distribution on the fly.
Datasets
The authors evaluated TempQuant across several challenging benchmarks:
- Reasoning: GSM8K (grade school math), MATH-500, AIME-120 (advanced math), and GPQA-Diamond (graduate-level science).
- Summarization: XSum.
- Models: Qwen3-4B/8B, Gemma 4 31B, and R1-Distill-Llama-8B, quantized using GPTQ (3-bit) and MXFP4 (4-bit).
Key Insights and Ideas
- Quantization makes models "hotter": Quantized models generally have higher entropy than their full-precision versions. Their probability mass is more spread out, making them act as if they were sampled at a higher temperature. Quantized models run hotter
- The Warming/Cooling Trade-off: The authors discovered a crucial split in token behavior. Tokens where the top-ranked choice remains the same after quantization ("preserved") usually need cooling to sharpen the distribution. However, tokens where the top choice changes ("flipped") carry most of the error and actually need warming to recover the original model's preferred token. Flipped tokens carry most of the error
- Divergence Choice is Task-Specific:
- Forward KL (mass-covering) is best for reasoning tasks because it prioritizes "warming" flipped tokens, which helps break repetitive loops.
- Reverse KL (mode-seeking) is better for short text and summarization because it prioritizes "cooling" to maintain the model's focus on high-probability tokens.
Key Results
- Accuracy Recovery: On 3-bit Qwen3-4B, Global TempQuant boosted MATH accuracy from 34% to 51% and GSM8K from 18% to 81%.
- Mimicking Manual Tuning: Global TempQuant finds nearly the same optimal settings as expensive manual tuning on labeled answers, but it does so using only a few hundred unlabeled responses.
- Behavioral Preservation: Beyond just accuracy, TempQuant brings the quantized model's overall behavior (repetition rates, word overlap) much closer to the original full-precision model.
Limitations and Future Work
- Ranking Limits: Temperature and top-p scaling cannot reorder tokens. If quantization moves the correct token from rank 1 to rank 100, these methods can only make it more likely to be picked; they can't make it rank 1 again.
- Runtime Overhead: While Global TempQuant is free, the current unoptimized per-step implementation adds 18–34% to the time per generated token.
- Domain Specificity: The controllers in this study were trained primarily on mathematical text. Future work could explore how well these learned settings generalize across more diverse datasets like code or creative writing.
Summary updated