Temperature scaling is a ubiquitous technique used to control the generation behavior—balancing "creativity" and "certainty"—in discrete sequence models like Large Language Models (LLMs). Traditionally, this is done by adjusting a scalar $T$ at the token level during inference. However, this "myopic" token-by-token approach induces bias in maximum a posteriori (MAP) estimates and fails to account for the probability distribution of the entire sequence.
This issue is further exacerbated in discrete diffusion models. In these architectures, a single sequence can be generated through many different "trajectories" (orders of unmasking tokens). Calculating the true likelihood of a sequence requires summing over all these paths, making likelihood computation computationally explosive and highly variant.
To address these limitations, this paper introduces a unified framework for sequence-level temperature scaling, utilizing novel fine-tuning objectives to achieve precise control over model sharpness without relying on intractable normalization constants.
A Unified Framework: SSUMs and Partition-Free Metrics
The researchers introduce Stochastic Sequence Unmasking Models (SSUMs), a novel formal notation that treats both standard autoregressive models (like GPT) and discrete diffusion models as a single mathematical class. This allows for a consistent sequence-level scaling approach across fundamentally different architectures.
The core mathematical hurdle of sequence-level scaling is the "partition function"—the sum of probabilities of all possible sequences, which is impossible to compute for natural language. The authors bypass this by utilizing partition-free metrics based on likelihood ratios between two sequences. By focusing on how the ratio of two sequences should change under a specific temperature, the impossible-to-calculate constants neatly cancel out.
Fine-Tuning Objectives for Sequence-Level Scaling
Instead of merely adjusting a parameter at inference time, the authors propose fine-tuning the model to "learn" the desired temperature. They introduce three primary loss functions:
- $L_{SSTS}$: An importance-sampling approach that is effective but can suffer from high variance.
- $L_p$ (Entropy Variance Regularization): A contrastive loss that encourages the model to maintain a constant ratio between its original and scaled probabilities. This objective is particularly effective at stabilizing diffusion models by reducing "schedule variance."
- $L_k$ (Monte-Carlo Regularization): A scalable $k$-sample estimator variant of $L_p$ designed to handle the high variance of diffusion trajectories.
Experimental Results: Calibration, Perplexity, and Reasoning
The proposed methods were evaluated across datasets like OpenWebText, MATH/MATH500, and HumanEval, using models including GPT-2, BD3LM, and Phi-3.5-mini-instruct.
- Precise Calibration: Using the $L_p$ objective, the models achieved "Effective Temperatures" ($T_{eff}$) that closely matched target temperatures (e.g., $T=0.1$ or $T=3.0$). Traditional myopic scaling proved "flat-broke" in experiments, struggling significantly when attempting to "flatten" (diversify) the distribution.
- Lower Perplexity: Fine-tuned models exhibited significantly lower Negative Log-Likelihood (NLL). For instance, at low temperatures, the sequence-level approach on GPT-2 achieved an NLL of 3.33, compared to 5.50 for the myopic baseline.
- Enhanced Reasoning: Sequence-level scaling demonstrably improved complex multi-step problem-solving. On the MATH500 benchmark, Phi-3.5-mini-instruct improved from a base accuracy of 33.0% to 45.2%.
Limitations and Future Directions
While highly effective, the authors note that relying solely on the temperature objective can cause "likelihood drift," where the model begins assigning low probabilities globally. This requires careful balancing with a standard language modeling loss. Additionally, while dynamic programming improves efficiency for block diffusion, computational cost still scales exponentially with block size ($2^B$).
Looking forward, this sequence-level framework holds significant promise for non-textual sequences, such as molecular generation, where precise probability control is critical for applications like drug discovery.
Summary updated