TempQuant: Learning Adaptive Temperatures for Quantized Language Models

Mukund Athmaram, Liang Ma, Manoj RaoarXiv 2026 · arXiv

TL;DR

TempQuant is a lightweight method to recover the accuracy lost when quantizing Large Language Models (LLMs) by automatically tuning decoding parameters like temperature and top-p. By matching the quantized model's probability distributions to its full-precision counterpart, it can significantly boost performance—for example, raising 3-bit Qwen3-4B’s MATH accuracy from 34% to 51%—without requiring any labeled data or model retraining.

What Problem Does This Paper Address?

Quantization (reducing the precision of model weights to 3 or 4 bits) is essential for running massive LLMs on consumer hardware, but it often degrades accuracy. Standard practice is to simply use the same decoding settings—specifically sampling temperature ($\tau$) and top-p ($\rho$)—as the original full-precision model.

The authors show this is a mistake. Quantization distorts the model's next-token probability distributions, effectively "warming" the model and increasing its output entropy (randomness). This can lead to repetitive loops or nonsensical outputs. The paper investigates whether adjusting these simple "knobs" at the decoding stage can undo the damage caused by quantization.

Figure 1: Illustrative three-token distributions before and after temperature and top-p correction.

Methods

The core idea of TempQuant is to find new temperature and top-p settings that minimize the Kullback–Leibler (KL) divergence between the quantized model and the original full-precision (BF16) model. They propose two versions:

  1. Global TempQuant: This finds a single optimal $(\tau, \rho)$ pair for the entire model for a given requested temperature. It uses a simple grid search on a small set of unlabeled data. Because it only changes two numbers in the sampler, it adds zero cost to deployment.
  2. Per-step TempQuant: This uses a tiny MLP "head" (only ~165k parameters) to predict a custom temperature correction and top-p cutoff for every single token generated. It looks at features of the quantized logits (like entropy and gaps between top tokens) to decide how to adjust the distribution on the fly.

Datasets

The authors evaluated TempQuant across several challenging benchmarks:

  • Reasoning: GSM8K (grade school math), MATH-500, AIME-120 (advanced math), and GPQA-Diamond (graduate-level science).
  • Summarization: XSum.
  • Models: Qwen3-4B/8B, Gemma 4 31B, and R1-Distill-Llama-8B, quantized using GPTQ (3-bit) and MXFP4 (4-bit).

Key Insights and Ideas

  • Quantization makes models "hotter": Quantized models generally have higher entropy than their full-precision versions. Their probability mass is more spread out, making them act as if they were sampled at a higher temperature. Quantized models run hotter
  • The Warming/Cooling Trade-off: The authors discovered a crucial split in token behavior. Tokens where the top-ranked choice remains the same after quantization ("preserved") usually need cooling to sharpen the distribution. However, tokens where the top choice changes ("flipped") carry most of the error and actually need warming to recover the original model's preferred token. Flipped tokens carry most of the error
  • Divergence Choice is Task-Specific:
  • Forward KL (mass-covering) is best for reasoning tasks because it prioritizes "warming" flipped tokens, which helps break repetitive loops.
  • Reverse KL (mode-seeking) is better for short text and summarization because it prioritizes "cooling" to maintain the model's focus on high-probability tokens.
Figure 2: Quantized models against BF16: (a) the BF16 temperature with the same mean entropy as GPTQ3, MXFP4, and BF16 with random noise added; (b) GPTQ3’s entropy change from BF16 at τ = 0.6, grouped by the average entropy of the two models, with each group’s share of tokens above.
Flipped tokens carry most of the error

Key Results

  • Accuracy Recovery: On 3-bit Qwen3-4B, Global TempQuant boosted MATH accuracy from 34% to 51% and GSM8K from 18% to 81%.
  • Mimicking Manual Tuning: Global TempQuant finds nearly the same optimal settings as expensive manual tuning on labeled answers, but it does so using only a few hundred unlabeled responses.
  • Behavioral Preservation: Beyond just accuracy, TempQuant brings the quantized model's overall behavior (repetition rates, word overlap) much closer to the original full-precision model.

Limitations and Future Work

  • Ranking Limits: Temperature and top-p scaling cannot reorder tokens. If quantization moves the correct token from rank 1 to rank 100, these methods can only make it more likely to be picked; they can't make it rank 1 again.
  • Runtime Overhead: While Global TempQuant is free, the current unoptimized per-step implementation adds 18–34% to the time per generated token.
  • Domain Specificity: The controllers in this study were trained primarily on mathematical text. Future work could explore how well these learned settings generalize across more diverse datasets like code or creative writing.

Summary updated

Abstract

Large language models are increasingly deployed at low precision, yet quantization can degrade their performance. We study how quantization interacts with decoding settings such as the sampling temperature and top-p by examining how it distorts the model's next-token probabilities. Measuring entropy during generation across deployed temperatures, we find that quantization warms the model on average, and that temperature matters much more for the performance of a quantized model than for its full-precision counterpart. Because quantization and temperature both shape the next-token distribution, we investigate how much the sampling settings can undo what quantization changes, through our method TempQuant, which learns the quantized model's temperature and top-p to bring its next-token probabilities closer to the full-precision model's. We learn these settings at two granularities. The first is a single global setting per requested temperature, which finds nearly the same temperatures that tuning on labeled answers would, without any labels. Model cards typically recommend sampling a quantized model with the same settings as its full-precision counterpart, while we find that different settings can substantially improve its accuracy. On 3-bit Qwen3-4B, for example, global TempQuant raises MATH accuracy from 34% under the configuration recommended for Qwen's quantized releases to 51%. The second sets the temperature and top-p for every token, motivated by our finding that tokens whose rankings flip under quantization tend to need warming, while tokens whose rankings are preserved tend to need cooling. When quantization degrades accuracy substantially, TempQuant recovers much of it, and when degradation is lighter, it still brings the quantized model's behavior closer to the full-precision model's.
BibTeX
@misc{athmaram2026tempquant,
  author = {Mukund Athmaram and Liang Ma and Manoj Rao},
  title = {{TempQuant: Learning Adaptive Temperatures for Quantized Language Models}},
  year = {2026},
  eprint = {2611.22128},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22128},
}
Figure