Large Language Models (LLMs) are notoriously prone to "hallucinations"—generating confident-sounding but factually incorrect text. While a model might explicitly state it is "99% sure" of an answer, these verbalized self-evaluations are often poorly calibrated and easily manipulated by prompting. A growing body of research suggests that the key to understanding what a model actually "knows" lies not in its textual output, but in its internal probabilistic state: the raw scores (logits) and token distributions generated during inference.
The Limits of Verbalized Confidence
When we ask an LLM to rate its own confidence, we are asking it to perform a meta-cognitive task that it wasn't explicitly trained for. As Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar argue in Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation, a model's internal probability distribution is a far more reliable indicator of correctness than its verbalized self-reports.
The problem with verbalized confidence is twofold. First, models often suffer from systematic overconfidence, a trait Hu Guo and colleagues address with the SMART recalibration method. Second, models may struggle to capture the nuance of human disagreement, as noted by James Thorne in Can Large Language Models Infer and Disagree Like Humans?. By looking at the "white-box" signals—the logits—researchers can bypass the unreliable layer of natural language.
Logits as a Proxy for Knowledge
At each step of generation, an LLM produces a vector of "logits"—unnormalized scores for every possible token in its vocabulary. These logits are typically passed through a softmax function to create a probability distribution. However, the raw logits themselves contain crucial information that softmax destroys.
In Semantic Energy: Detecting LLM Hallucination Beyond Entropy, researchers demonstrate that the absolute magnitude of raw logits serves as a proxy for the model's exposure to specific facts during training. High-intensity logits suggest the model is operating within a high-density region of its training data (epistemic certainty), while low-intensity logits—even if they result in a "clear" top choice after softmax—signal that the model is guessing. This "introspection" approach was pioneered by Marcin Detyniecki in Detecting Adversarial Examples and Other Misclassifications in Neural Networks by Introspection, showing that secondary models can "read" these internal activations to predict errors.
Measuring the Decision Margin
A recurring theme across this research is that the relationship between the top predicted tokens is often more informative than the absolute probability of the winner.
- The Logit Gap: Hu Guo (SMART) and Yucheng Suo (From Trial to Triumph) both emphasize the "marginal confidence"—the difference between the highest and second-highest logit. A large gap indicates the model has a clear "winner," while a small gap suggests the sample lies near a decision boundary.
- Entropy and Concentration: Hao Yang and Qi Zhao show in Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation that effective reasoning (like CoT) works by "concentrating" the probability distribution, making the final answer token significantly more likely than the alternatives.
- Predictive Entropy: In the ALIGN system for medical coding, Nabeel Seedat uses entropy across multiple-choice reformulations to identify when a model should defer to a human expert.
Adaptive Inference and Cost Control
Beyond error detection, these internal signals enable "adaptive inference"—the ability to adjust computational effort based on task difficulty.
The AdaInfer algorithm by Siqi Fan monitors logit stability across transformer layers. If the top-token probability and the logit margin stabilize early, the model can "exit" the inference process, saving up to 43% of compute. Similarly, Sang-Mook Lee introduces STEER, which uses logit-based confidence to route easy reasoning steps to small models and difficult ones to large models.
This concept is extended to "thinking" models in ThinkBrake by Minjae Oh. By monitoring the log-probability of the "stop thinking" token, the system can prevent "overthinking"—where a model reaches the correct answer in its chain-of-thought but eventually talks itself into an error.
Applications in High-Stakes Domains
The ability to extract reliable uncertainty from logits is transformative for specialized fields:
- Medicine: Harsha Nori and Eric Horvitz demonstrate in Capabilities of GPT-4 on Medical Challenge Problems that token-level probabilities allow GPT-4 to achieve near-perfect calibration on medical exams.
- Chemistry: Mayk Caldas Ramos and Andrew Dickson White use token probabilities to drive Bayesian Optimization of Catalysts, treating the LLM as a surrogate model that provides both predictions and the uncertainty estimates needed for exploration.
- Robotics: Thomas P Zollo and Richard Zemel apply these techniques to Vision-Language-Action models, aggregating action token probabilities to help robots know when they are likely to fail a task.
By shifting focus from what LLMs say to the statistical properties of their internal predictions, researchers are building a more robust, efficient, and "self-aware" generation of AI systems.