Temperature Scaling in Discrete Sequence (Language) Models

Hannah Scheufele, Peter Blohm, Vikas K GargICML 2026 · OpenReview

Temperature scaling is a ubiquitous technique used to control the generation behavior—balancing "creativity" and "certainty"—in discrete sequence models like Large Language Models (LLMs). Traditionally, this is done by adjusting a scalar $T$ at the token level during inference. However, this "myopic" token-by-token approach induces bias in maximum a posteriori (MAP) estimates and fails to account for the probability distribution of the entire sequence.

This issue is further exacerbated in discrete diffusion models. In these architectures, a single sequence can be generated through many different "trajectories" (orders of unmasking tokens). Calculating the true likelihood of a sequence requires summing over all these paths, making likelihood computation computationally explosive and highly variant.

To address these limitations, this paper introduces a unified framework for sequence-level temperature scaling, utilizing novel fine-tuning objectives to achieve precise control over model sharpness without relying on intractable normalization constants.

A Unified Framework: SSUMs and Partition-Free Metrics

The researchers introduce Stochastic Sequence Unmasking Models (SSUMs), a novel formal notation that treats both standard autoregressive models (like GPT) and discrete diffusion models as a single mathematical class. This allows for a consistent sequence-level scaling approach across fundamentally different architectures.

The core mathematical hurdle of sequence-level scaling is the "partition function"—the sum of probabilities of all possible sequences, which is impossible to compute for natural language. The authors bypass this by utilizing partition-free metrics based on likelihood ratios between two sequences. By focusing on how the ratio of two sequences should change under a specific temperature, the impossible-to-calculate constants neatly cancel out.

Fine-Tuning Objectives for Sequence-Level Scaling

Instead of merely adjusting a parameter at inference time, the authors propose fine-tuning the model to "learn" the desired temperature. They introduce three primary loss functions:

  • $L_{SSTS}$: An importance-sampling approach that is effective but can suffer from high variance.
  • $L_p$ (Entropy Variance Regularization): A contrastive loss that encourages the model to maintain a constant ratio between its original and scaled probabilities. This objective is particularly effective at stabilizing diffusion models by reducing "schedule variance."
  • $L_k$ (Monte-Carlo Regularization): A scalable $k$-sample estimator variant of $L_p$ designed to handle the high variance of diffusion trajectories.

Experimental Results: Calibration, Perplexity, and Reasoning

The proposed methods were evaluated across datasets like OpenWebText, MATH/MATH500, and HumanEval, using models including GPT-2, BD3LM, and Phi-3.5-mini-instruct.

  • Precise Calibration: Using the $L_p$ objective, the models achieved "Effective Temperatures" ($T_{eff}$) that closely matched target temperatures (e.g., $T=0.1$ or $T=3.0$). Traditional myopic scaling proved "flat-broke" in experiments, struggling significantly when attempting to "flatten" (diversify) the distribution.
  • Lower Perplexity: Fine-tuned models exhibited significantly lower Negative Log-Likelihood (NLL). For instance, at low temperatures, the sequence-level approach on GPT-2 achieved an NLL of 3.33, compared to 5.50 for the myopic baseline.
  • Enhanced Reasoning: Sequence-level scaling demonstrably improved complex multi-step problem-solving. On the MATH500 benchmark, Phi-3.5-mini-instruct improved from a base accuracy of 33.0% to 45.2%.

Limitations and Future Directions

While highly effective, the authors note that relying solely on the temperature objective can cause "likelihood drift," where the model begins assigning low probabilities globally. This requires careful balancing with a standard language modeling loss. Additionally, while dynamic programming improves efficiency for block diffusion, computational cost still scales exponentially with block size ($2^B$).

Looking forward, this sequence-level framework holds significant promise for non-textual sequences, such as molecular generation, where precise probability control is critical for applications like drug discovery.

Summary updated

Abstract

Temperature scaling (TS) is widely used to control generation behavior in discrete sequence models. However, TS at the sequence level is challenging as the needed partition function is typically intractable and naive *token-wise* TS induces bias. Discrete diffusion settings exacerbate this issue: enhanced flexibility due to multiple potential orderings for generating tokens complicates likelihood computation and incurs high variance. We propose and investigate fine-tuning objectives for sequence-level TS, along with metrics to quantify the success of a given TS procedure. Specifically, our contributions are threefold. First, we introduce our objective across a wide class of sequence models through a unifying formalism. Second, we design two metrics that quantify TS based on likelihood *ratios* and thus obviate access to the partition function. Finally, we introduce fine-tuning objectives that reliably achieve the desired change in model temperature. Our experiments with language models (GPT2, BD3LM) show that the proposed approach leads to more consistent generation with lower Negative Log-Likelihood (NLL). Furthermore, we provide empirical evidence that it can enhance the reasoning performance in language models. Code is available at [github.com/Aalto-QuML/temp_lm](https://github.com/Aalto-QuML/temp_lm).
BibTeX
@inproceedings{scheufele2026temperature,
  author = {Hannah Scheufele and Peter Blohm and Vikas K Garg},
  title = {{Temperature Scaling in Discrete Sequence (Language) Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=bHIeH7450V},
}
Figure