Semantic Entropy for Language Model Uncertainty Estimation

407 concepts221 papers1 proposal

The Token-Level Fallacy

When evaluating whether a Large Language Model (LLM) "knows" a fact or is simply guessing, a natural first instinct is to look at its internal token probabilities. If a model assigns a high probability to a specific sequence of tokens, we might assume it is confident. Conversely, if the probability mass is spread thinly across many different token sequences, standard information theory suggests high entropy, which we typically interpret as uncertainty.

However, as highlighted by Lorenz Kuhn et al. in Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation, this token-level view suffers from a fundamental flaw: semantic multiplicity. Natural language allows the same underlying concept to be expressed through countless distinct word sequences.

Consider a model asked: "Who wrote Hamlet?" The model might generate three distinct responses across different sampling trials:

  1. "William Shakespeare"
  2. "Shakespeare"
  3. "The Bard of Avon"

Lexically, these strings share very few tokens. If we compute standard token-level or sequence-level entropy, the model appears highly erratic and uncertain. Yet, semantically, the model is completely certain—all three answers point to the exact same historical figure. Traditional entropy metrics penalize the model for its linguistic creativity, misclassifying stylistic variation as a lack of knowledge.

To solve this, a growing body of research shifts the focus from lexical uncertainty to semantic uncertainty.

The Mechanism of Semantic Clustering

To measure true uncertainty, we must marginalize out superficial syntactic variations. The standard framework for doing this involves a three-step pipeline: sampling, clustering, and aggregation.

First, the model is prompted multiple times (often using temperature sampling) to generate a set of candidate responses.

Second, these responses are grouped into semantic equivalence classes. To teach an algorithm to recognize that two entirely different strings mean the same thing, the most common approach uses a secondary Natural Language Inference (NLI) model to evaluate mutual semantic entailment. If Response A logically implies Response B, and Response B logically implies Response A, they are grouped into the same cluster.

For example, Ruiyang Zhang et al. in VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation utilize mutual entailment to cluster responses in vision-language tasks, ensuring that the model isn't penalized for describing an image using different but accurate synonyms.

Third, once the responses are clustered, the probability mass is aggregated. Instead of treating each string $s$ as an independent outcome, the probability of a semantic cluster $C$ is defined as the sum of the probabilities of all sequences belonging to that cluster:

$$P(C) = \sum_{s \in C} P(s)$$

Finally, Semantic Entropy is computed as the Shannon entropy over these cluster probabilities rather than individual token sequences. If all generated responses collapse into a single semantic cluster, the semantic entropy is zero, signaling high confidence, regardless of how many different words were used. If the responses split into fundamentally contradictory clusters (e.g., one cluster asserting "True" and another "False"), the semantic entropy spikes, indicating genuine conceptual confusion.

Overcoming the Black-Box Limitation

The mathematical formulation of semantic entropy assumes we have access to the model's internal token probabilities ($P(s)$) to sum them up. However, many state-of-the-art LLMs are hosted behind commercial APIs that hide these internal logits. To estimate semantic uncertainty when the model is a complete "black box", several creative proxies have emerged:

  • Graph-Based Eccentricity: As implemented in the LM-Polygraph framework by Ekaterina Fadeeva et al., sampled responses can be represented as nodes in a semantic network. By measuring the distance or "eccentricity" of these nodes based on semantic similarity, the system can detect outlier interpretations without needing internal model parameters.
  • Semantic Alphabet Size: Instead of calculating precise probabilities, Lucas Hurley McCabe et al. propose a simplified proxy in Estimating Semantic Alphabet Size for LLM Uncertainty Quantification. Borrowing techniques from population ecology, the authors estimate the total number of unique semantic categories (the "alphabet size") a model could generate from a small sample. They demonstrate that the sheer diversity of unique meanings is often a strong enough signal to detect hallucinations on its own.
  • Semantic Density Estimation: Rather than hard-clustering responses, the authors of Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space map discrete textual outputs into a continuous semantic vector space. They apply kernel density estimation to measure the concentration of high-probability responses around a target output, providing a robust confidence signal that generalizes across diverse architectures.

From Estimation to Mitigation

Quantifying uncertainty is not just a diagnostic exercise; it provides a powerful signal for improving model behavior.

One major application is teaching models when to safely abstain from answering. In Fine-Tuning Large Language Models to Appropriately Abstain with Semantic Entropy, Benedict Aaron Tjandra et al. incorporate semantic entropy directly into the fine-tuning loop. By identifying prompts where the model's sampled meanings diverge wildly, the authors train the model to output phrases like "I don't know" or "I am unsure," significantly reducing hallucination rates without requiring expensive human-annotated datasets.

Similarly, semantic consistency can act as an unsupervised reward signal. In Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization, Qingyang Zhang et al. use semantic clustering to optimize reasoning paths. By minimizing semantic entropy across multiple internal trials, the model learns to favor paths that lead to a stable, consensus conclusion, boosting mathematical and logical reasoning accuracy without any external golden labels.

By shifting our mathematical lens from the surface level of syntax to the deeper layer of semantics, we gain a far more accurate, robust, and theoretically sound method for auditing what large language models truly know.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 210 papers
  2. Version 1 · 2026-10-04 · 311 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_uncertainty_estimation_dea50a6e,
  title = {semantic uncertainty estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-uncertainty-estimation/rdr_dea50a6e2ba242eeb03f5f7b31e9b8c5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 2, frozen text):

@misc{lacuna_semantic_uncertainty_estimation_dea50a6e_v2,
  title = {semantic uncertainty estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-uncertainty-estimation/rdr_dea50a6e2ba242eeb03f5f7b31e9b8c5/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake