Semantic Uncertainty and Calibration for Large Language Models

This direction is split into other directions: long-form uncertainty estimation, open-ended generation uncertainty. Its last version stays readable and citable here.

Version 1 of 1453 papers503 conceptsfrozen 2026-10-04

Large Language Models (LLMs) are notoriously "confidently wrong." They can generate fluent, authoritative prose that is factually untethered—a phenomenon known as hallucination. The core of this problem lies in a mismatch between a model's internal certainty and its external expression. While traditional machine learning relies on "logits" (raw probability scores) to estimate confidence, these scores often fail for LLMs, especially after they have been fine-tuned with human feedback, which tends to push models toward overconfidence.

Recent research has shifted from measuring the probability of specific words to measuring the consistency of meanings. This transition from token-level probability to semantic uncertainty is the key to building models that know when they don't know.

From Tokens to Meanings

The most intuitive way to measure uncertainty is to look at the model's output distribution. If a model assigns a 90% probability to the word "Paris," it seems certain. However, language is redundant. A model might split its probability between "Paris," "The capital of France," and "Lutetia." In token space, the model looks uncertain (low probability for any single string); in semantic space, it is perfectly certain.

To solve this, Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar introduced Semantic Entropy. Instead of calculating entropy over raw tokens, they cluster model outputs based on semantic equivalence. If the model generates ten different sentences that all mean the same thing, its semantic entropy is low, and its reliability is high.

This concept is extended by frameworks like LUQ (Caiqi Zhang) and LofreeCP (Jiayuan Su), which measure consistency across multiple sampled responses. The intuition is simple: if you ask a model the same question five times and get five different answers, the model is likely hallucinating.

Verbalizing Confidence

If we want to know how confident a model is, why not just ask it? This is the premise of "verbalized confidence." Eric Mitchell and colleagues demonstrated in Just Ask for Calibration that simply prompting a model to state its confidence in plain text (e.g., "I am 80% sure") can be more accurate than looking at its internal math.

This works because high-capacity models like GPT-4 have a "meta-cognitive" sense of their own knowledge boundaries. Researchers have refined this through methods like COKE (Lida Chen), which teaches models to recognize their knowledge boundaries using their own internal signals, and UaIT (Shudong Liu), which uses self-training to align a model's verbalized confidence with its actual accuracy.

Statistical Guarantees with Conformal Prediction

For high-stakes applications like medicine, "feeling" confident isn't enough; we need mathematical guarantees. This is where Conformal Prediction (CP) comes in. CP allows a model to output a set of possible answers rather than a single one, guaranteeing that the correct answer is in that set with a user-specified probability (e.g., 95%).

While CP is traditionally used for simple classification, Bhawesh Kumar and others have applied it to Multi-Choice Question Answering. More advanced frameworks like TRON (Bo Fu) and Conformal Arbitrage (William Overman) extend these ideas to "black-box" models where we only have API access, providing a way to bound error rates without seeing the model's internal weights.

The Frontier of Long-Form Generation

The hardest challenge in uncertainty quantification is long-form text. A three-paragraph biography might be 90% accurate but contain one critical hallucinated date. Standard metrics that provide a single score for the whole response fail to capture this nuance.

To address this, Mingjian Jiang and Tatsunori Hashimoto proposed Graph-based Uncertainty Metrics. They treat the relationship between a model's response and its individual claims as a bipartite graph. By measuring the "closeness centrality" of claims—essentially how well a claim is supported by other variations of the model's output—they can pinpoint exactly which sentence in a paragraph is likely a hallucination.

Teaching Models to Say I Don't Know

Ultimately, the goal of this research is to move from detecting uncertainty to acting on it. Frameworks like RLKF (Zichen Zhu) use reinforcement learning to teach models to explicitly refuse questions that exceed their knowledge. Similarly, Qi Yang and colleagues showed that Uncertainty-aware Language Modeling allows models to "abstain" from answering, achieving near-perfect accuracy by only speaking when they are certain.

By shifting the focus from word probabilities to semantic consistency and knowledge boundaries, researchers are creating a path toward LLMs that are not just capable, but trustworthy.

Papers in this version

Showing 60 of 453 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_uncertainty_and_calibration_for_a6221972,
  title = {Semantic Uncertainty and Calibration for Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-uncertainty-and-calibration-for-large-language-models/rdr_a6221972d0d640609fcf81656463d957},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_uncertainty_and_calibration_for_a6221972_v1,
  title = {Semantic Uncertainty and Calibration for Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-uncertainty-and-calibration-for-large-language-models/rdr_a6221972d0d640609fcf81656463d957/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}