When a machine learning model classifies an image, it chooses from a fixed, predefined menu of options. Quantifying its uncertainty is straightforward: we look at the probability distribution over those few classes. If the top class has a low probability, or if the entropy across classes is high, the model is uncertain.
Large Language Models (LLMs) operating in open-ended generation tasks—such as summarization, dialogue, or long-form reasoning—do not have a fixed menu. The output space is the set of all possible token sequences, which is effectively infinite. This structural shift breaks traditional uncertainty quantification (UQ) in two profound ways:
- The Semantic Dispersion Problem: A model can express the exact same meaning using entirely different words (e.g., "Yes, that is correct" vs. "Indeed, you are right"). If a model distributes its probability mass across fifty synonymous phrases, token-level logits will make the model appear highly uncertain, even though it is completely certain on a semantic level.
- The Infinite Enumeration Problem: Rigorous statistical frameworks like conformal prediction require calculating scores across all possible labels to guarantee that the true answer is contained within a predicted set. In open-ended text generation, enumerating every possible string is computationally impossible.
To resolve these challenges, a major research direction has emerged to move uncertainty quantification away from token-level probabilities and into semantic space, while developing novel statistical bounds capable of handling infinite, unstructured outputs.
Shifting from Token Logits to Semantic Space
To overcome semantic dispersion, recent techniques analyze the consistency of meaning across multiple sampled outputs rather than the raw probabilities of individual tokens. The intuition is simple: if we prompt a model multiple times and it generates structurally diverse but semantically identical answers, it possesses high epistemic certainty.
Xin Qiu et al. demonstrate this intuition in Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space by measuring confidence via semantic density. Similarly, Jiayuan Su et al. introduce LofreeCP in API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access, showing that combining response frequency with semantic similarity allows for effective uncertainty estimation even when internal model logits are completely inaccessible.
Taking this a step further, Yukun Li et al. propose a graph-based approach in Graph-based Confidence Calibration for Large Language Models. By constructing a graph where nodes represent different generated responses and edges represent semantic agreement patterns, the authors use Graph Neural Networks to accurately predict whether an answer is correct based purely on its topological consensus within the sample pool.
[Prompt] ---> (Sample 1: "It's correct.") <-- Semantic --> (Sample 2: "That is right.")
---> (Sample 3: "Paris is blue.") <-- Outlier --> High Semantic Uncertainty
Conformal Prediction in Infinite Output Spaces
Conformal prediction provides mathematical guarantees that a model's prediction set will contain the true answer with a user-defined probability (e.g., 95%). Translating this to free-form text requires bypassing the need to evaluate every possible sequence.
The breakthrough lies in calibrating a dynamic sampling and rejection process instead of an exhaustive search. In Conformal Language Modeling, Victor Quach et al. present a framework that provides correctness guarantees for generated sets of responses or specific sentences by bounding the generation space. To make these sets practically useful, Jinhao Duan et al. introduce ConU in Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees, which merges semantic self-consistency with conformal prediction to produce compact, statistically sound prediction sets.
However, a unique risk in open-ended generation is that the true, correct response might not appear anywhere within a finite pool of model samples. To address this "missing mass" problem, Sima Noorani et al. develop a framework in Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Models. Their approach uses a missing-mass perspective to rigorously bound the probability that the true answer lies entirely outside the generated candidate pool, providing elegant uncertainty bounds for open-ended tasks.
Black-Box Elicitation and Verbalized Confidence
When interacting with commercial LLMs via black-box APIs, practitioners cannot access internal embeddings or logits. In these scenarios, researchers look to verbalized confidence—forcing the model to evaluate its own certainty and express it directly in natural language (e.g., "Confidence: 80%").
Early work by Stephanie Lin et al. in Teaching Models to Express Their Uncertainty in Words demonstrated that models can be trained to verbalize well-calibrated probabilities that generalize across tasks, suggesting that models maintain latent internal representations of their own knowledge gaps. For frozen black-box models, Makoto Yamada et al. show in On Verbalized Confidence Scores for LLMs that while smaller models struggle with severe overconfidence, larger models can provide highly reliable verbalized scores when guided by strategic prompting.
To mitigate the inherent biases of a single verbalized response, researchers use ensembling and behavioral manipulation:
- Prompt Ensembling: Francesco Tonolini et al. introduce BayesPE in Bayesian Prompt Ensembles: Model Uncertainty Estimation for Black-Box Large Language Models, which uses a weighted ensemble of semantically equivalent prompts to capture model variance without architectural access.
- Fidelity Testing: Mozhi Zhang et al. propose UF Calibration in Calibrating the Confidence of Large Language Models by Eliciting Fidelity. They decompose confidence into uncertainty (variance across samples) and fidelity (how stubbornly the model sticks to its answer when the prompt options are intentionally manipulated).
- Strategic Aggregation: Miao Xiong et al. systematically evaluate these black-box strategies in Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs, confirming that clever aggregation narrows the performance gap between black-box elicitation and internal "white-box" logit methods.
Redefining Evaluation Metrics for Generative Calibration
Traditional calibration metrics like Expected Calibration Error (ECE) or Area Under the ROC Curve (AUROC) rely on a binary definition of correctness (true or false). In free-form text, a summary or dialogue response is rarely completely right or completely wrong; instead, its quality falls along a continuous spectrum.
To bridge this gap, Xinmeng Huang et al. introduce Rank-Calibration in Uncertainty in Language Models: Assessment through Rank-Calibration. Instead of forcing a binary correctness label, Rank-Calibration evaluates a fundamental requirement: whether a model's stated uncertainty consistently correlates with the multi-faceted quality of its generated text. If a model assigns higher uncertainty to a low-quality summary than to a high-quality summary, it is well-ranked, even if a binary error cannot be cleanly defined.
This shift in evaluation is vital. As Akim Tsvigun documents in LM-Polygraph: Uncertainty Estimation for Language Models, the reliability of uncertainty metrics degrades significantly as tasks transition from fact-based question answering to highly subjective, open-ended generation. By evaluating calibration as a ranking problem over semantic quality rather than a binary classification problem over tokens, these new frameworks allow for dependable uncertainty quantification in truly open-ended natural language settings.
Key Foundational Work
How Can We Know When Language Models Know? (2020) — contributed 2 concept elements to this direction.