Estimating Uncertainty From Internal Model Representations

128 concepts92 papers1 proposal

Deep learning models, particularly Large Language Models (LLMs), are notoriously overconfident. They frequently hallucinate incorrect facts or make erroneous predictions with absolute certainty. Traditionally, we evaluate a model's confidence by looking at its surface-level outputs: either the softmax probabilities of the generated tokens or explicit verbalized self-assessments (e.g., asking the model, "How sure are you?").

However, both approaches are deeply flawed. Softmax probabilities are often distorted by alignment techniques like Reinforcement Learning from Human Feedback (RLHF) or regularization methods like label smoothing, as noted by Johnathan Xie et al. [Paper] and Qiujia Li et al. [Paper]. Meanwhile, verbalized confidence scores are highly prone to text-level hallucinations, failing to reflect the model's actual underlying knowledge boundary, as shown by Shiyu Ni et al. [Paper] and Jiarui Liu et al. [Paper].

A rapidly growing body of research offers a more elegant solution: look inside the model. Instead of relying on the compressed, flattened information at the final output layer, we can treat the model's internal hidden states as a rich reservoir of latent uncertainty. The core intuition is simple: the model often "knows" it is guessing long before it finishes generating an answer.


The Core Mechanism: Probing the Latent Space

When a model processes an input, it constructs high-dimensional internal representations across dozens of layers. If a model lacks the knowledge required to answer a prompt, or if the prompt is inherently ambiguous, this deficit leaves a distinct mathematical signature within those hidden activations.

To extract this signal, researchers train lightweight, non-destructive classifiers—often called linear probes—directly on top of the model's intermediate hidden states.

[ Input Prompt ] ---> [ Layer 1 ] ---> [ Layer 2 ] ---> [ Layer N ] ---> [ Softmax Output ]
                                            |
                                            +---> [ Linear Probe ] ---> [ Reliable Confidence Score ]

This approach bypasses the need for expensive black-box sampling methods. For instance, while computing "Semantic Entropy" normally requires generating and clustering multiple independent text responses to find meaning-based variations (Lorenz Kuhn et al. [Paper]), Jannik Kossen et al. demonstrate that Semantic Entropy Probes (SEPs) can capture this exact same predictive power in a single forward pass by reading the internal states directly [Paper].


Where Does Uncertainty Live?

Uncertainty does not manifest uniformly across a neural network. Analyzing the layer-wise dynamics reveals how a model processes its own doubt:

  • Lower Layers (Structural & Contextual Information): Yu Lu et al. find that lower-layer hidden states often provide superior signals for uncertainty estimation compared to final layers [Paper]. While the final layers are heavily specialized for predicting specific tokens, the earlier layers preserve broader structural and semantic context that reflects underlying confidence.
  • Intermediate Layers (Disambiguation): Gustaf Ahdritz demonstrates that the hidden states of middle layers are highly effective at distinguishing between different types of uncertainty [Paper]. For example, a model's intermediate layers display distinct representational patterns when facing a structural lack of factual knowledge versus encountering an inherently ambiguous linguistic phrase.
  • Upper Layers (Distributional Bounds): In the top layers of the network, the entropy of the hidden states begins to correlate tightly with the final output. Shiqi Gao et al. show that upper-layer hidden state entropy serves as a theoretical upper bound for the entropy of the final predictive distribution, allowing systems to estimate output uncertainty without ever running the full decoding process [Paper].

Why It Works: The Shared Circuits Intuition

Why do these high-dimensional vectors contain such reliable indicators of truthfulness and doubt?

A foundational question in model interpretability is whether networks compute uncertainty via a specialized, parallel internal mechanism, or if it is an inherent property of standard information processing. Using mechanistic interpretability tools, Carter Teplica et al. revealed that uncertainty is computed via shared circuits [Paper].

When you break or strain the internal circuit responsible for retrieving a specific fact, you simultaneously disrupt the model's internal representation of confidence. In other words, uncertainty is not an afterthought; it is an emergent property of factual retrieval itself. When a model tries to retrieve a concept it doesn't possess, the activations within its feed-forward networks naturally scatter or degrade. This structural disruption can be quantified by measuring representational consistency across layers (Jiafeng Guo et al. [Paper]) or by analyzing the topological "shape" of the activation graphs (Théo Lacombe et al. [Paper]).

From a theoretical standpoint, Preetum Nakkiran proves that this "semantic calibration" emerges naturally from standard next-token loss optimization [Paper]. Because language models are trained to minimize cross-entropy over diverse concepts, their internal representations are mathematically incentivized to encode the true concept-level probabilities long before they are forced to pick a single word.


Applications Across Domains

By tapping into these latent signals, researchers have unlocked significant performance gains across several high-stakes domains:

1. Dynamic Calibration and Temperature Scaling

Instead of applying a fixed temperature to all outputs, systems can use internal hidden states as diagnostic features to predict an optimal, instance-specific temperature. Johnathan Xie et al. developed Adaptive Temperature Scaling (ATS) to adjust confidence token-by-token, drastically reducing calibration errors without hurting task performance [Paper].

2. Adaptive Tool Use in Language Agents

Language agents often invoke external tools (like search engines or calculators) indiscriminately. By training linear probes on internal activations, frameworks designed by Hao Liu et al. [Paper] and Wenjun Li et al. [Paper] allow models to evaluate their own competence first. If the internal state signals high certainty, the model answers immediately; if it signals doubt, the agent dynamically triggers the appropriate tool.

3. Generalization Beyond Text

The principle that internal latent tokens carry implicit difficulty and uncertainty signals extends far beyond natural language generation:

  • Vision Transformers (ViTs): Wenhao Liang et al. demonstrate that global summary tokens in ViTs naturally encode sample difficulty, which can be decoded to perform instance-wise calibration [Paper].
  • Object Segmentation: Timo Kaiser et al. show that training lightweight multi-layer perceptrons on the internal latent tokens of the Segment Anything Model (SAM) provides fast, efficient uncertainty quantification without requiring expensive Bayesian sampling [Paper].

Ultimately, representation-based uncertainty estimation shifts our relationship with deep learning models from treating them as black boxes to treating them as introspective systems. By listening to the model's internal representations rather than just its final words, we can build AI systems that are fundamentally more self-aware, honest, and reliable.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_internal_state_uncertainty_ba54d9a5,
  title = {internal state uncertainty},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-state-uncertainty/rdr_ba54d9a5aeb844b9be68ca197ebd5698},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_internal_state_uncertainty_ba54d9a5_v1,
  title = {internal state uncertainty},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-state-uncertainty/rdr_ba54d9a5aeb844b9be68ca197ebd5698/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake