When a machine learning model makes a high-stakes prediction—such as forecasting a clinical outcome or diagnosing a medical condition—we need to know how confident it is. Traditionally, we calculate this by looking "under the hood" at the model's internal state. By analyzing the logit distributions or softmax probabilities at the output layer, we can easily see if a model is decisively picking an answer or if it is torn between multiple choices.
However, many of the world's most powerful large language models (LLMs) are now deployed as closed-source, proprietary "black boxes." When interacting with these models via commercial APIs, users do not have access to internal token probabilities, gradients, or hidden states. The API accepts a text prompt and returns a text string.
This opacity breaks traditional uncertainty quantification (UQ). If a model outputs a factual claim, how can we determine whether it is drawing from a high-probability internal belief or hallucinating an obscure guess? To solve this, a new research direction focuses on black-box uncertainty quantification: estimating model reliability using nothing but the text-based interface itself.
The Core Intuition: Semantic Consistency
If you ask a knowledgeable human a question they are certain about (e.g., "What is the capital of France?"), they will repeatedly give you the same answer, even if they phrase it slightly differently ("Paris," "It's Paris," or "The capital is Paris"). If you ask them something they are highly uncertain about, their guesses will diverge wildly across different attempts.
This is the foundational intuition behind sampling-based black-box UQ. By setting the model's generation temperature higher, we can sample multiple independent responses to the same prompt. If the model is certain, these samples will cluster tightly together in semantic space. If the model is guessing or hallucinating, the samples will scatter.
Certainty: [Sample 1: Paris] ─── [Sample 2: Paris] ─── [Sample 3: It is Paris] (Tight Cluster)
Uncertainty: [Sample 1: London] ─── [Sample 2: Berlin] ─── [Sample 3: I don't know] (High Dispersion)
The challenge lies in measuring this dispersion. Because language is flexible, two answers can use entirely different words while meaning the exact same thing. Simple string matching fails.
To address this, Lin et al. proposed analyzing the semantic similarity and graph-based dispersion of multiple generated responses. Instead of looking at token-level math, they evaluate how much the meaning of the outputs fluctuates. Building on this concept, Nikitin et al. introduced Kernel Language Entropy (KLE), which uses graph kernels and quantum-inspired von Neumann entropy to measure fine-grained semantic uncertainties without needing internal log-probabilities. For long-form text where consistency must be tracked over extended passages, the authors of LUQ developed a framework to measure consistency at the sentence level across multiple sampled generations.
This semantic consistency approach has proven highly effective in specialized domains. For instance, Wang et al. applied it to radiology report generation, tracking the stability of clinical facts across samples to flag unreliable medical summaries. Similarly, the authors of a study on vision-language models utilized Discrete Semantic Entropy (DSE) to filter out hallucinations in multimodal radiology workflows, significantly boosting diagnostic accuracy by discarding high-entropy outputs.
Conformal Prediction Without Logits
While measuring semantic consistency gives an intuitive grasp of uncertainty, safety-critical applications often require rigorous statistical guarantees. This is typically the domain of Conformal Prediction (CP)—a framework that takes a model's prediction and constructs a prediction set guaranteed to contain the true answer with a user-defined probability (e.g., 95%).
Standard conformal prediction relies on sorting a model's internal logits to determine which tokens are likely enough to make the cut. In a black-box setting, researchers must find a proxy for these missing logits.
White-Box CP: [Internal Logits] ───────> [Sort & Threshold] ───> Valid Prediction Set
Black-Box CP: [Sampled Text Outputs] ──> [Proxy Score (Freq)] ──> Valid Prediction Set
Su et al. solved this with LofreeCP, a framework that constructs valid prediction sets by substituting logit access with a combination of generation frequencies, semantic similarity, and text-based entropy. By treating the empirical frequency of sampled answers as a proxy for probability, they achieve rigorous statistical guarantees purely through an API.
Similarly, Li et al. introduced TRAQ to bring these end-to-end correctness guarantees to Retrieval-Augmented Generation (RAG) pipelines, constructing reliable answer sets based entirely on the distribution and semantic overlap of black-box samples. This shift from internal math to external sampling allows practitioners to benchmark and bound the uncertainty of proprietary models systematically, as demonstrated by Ye et al. in broad LLM uncertainty benchmarking.
Verbalized Confidence and Behavioral Probing
If an LLM is designed to simulate human-like conversation, why not simply ask it how confident it is? This approach is known as verbalized confidence elicitation.
User: "What is the capital of Australia?"
LLM: "Canberra."
User: "On a scale of 0 to 100, how confident are you in this answer?"
LLM: "95."
Early evaluations of this method revealed a major flaw: language models are naturally overconfident, often mimicking human bravado rather than factual accuracy. Yuan et al. demonstrated that models like ChatGPT exhibit significant overconfidence when their internal "semantic priors" conflict with the specific rules of a task. Furthermore, Madhusudhan et al. showed that models struggle with abstention ability—knowing when not to answer—frequently hallucinating instead of admitting ignorance on niche topics.
However, verbalization can be unlocked with the right prompting and aggregation strategies. Yamada et al. discovered a strong scaling effect: while small models are hopelessly miscalibrated, larger models can provide highly accurate verbalized confidence scores if guided by strategic prompting. To mitigate individual prompt biases, the authors of BayesPE introduced Bayesian Prompt Ensembles, which use a weighted ensemble of semantically equivalent prompts to get a well-calibrated, model-agnostic uncertainty estimate.
Taking this a step further, Xiong et al. showed that combining verbalized confidence with response sampling and clever aggregation can narrow the performance gap between black-box methods and white-box methods.
Active Probing: Follow-up Queries
Rather than relying on a single self-reported number, some frameworks treat uncertainty estimation as an active interrogation.
Sam et al. developed QueRE (Follow-up Question Representation Elicitation). Instead of trusting the initial answer, the system hits the black-box model with a series of standardized follow-up questions, such as "Are you sure about that?" or "Can you double-check your reasoning?" By analyzing the text responses to these challenges, the framework builds a feature vector that accurately predicts whether the original answer was correct.
In a similar vein of behavioral probing, Wen et al. designed a framework to map an LLM's "knowledge boundary" using semi-open-ended questions. By using a smaller, open-source auxiliary model to systematically "poke" the black-box model, they expose exactly where a model's factual competence ends and where its overconfident hallucinations begin.
Summary
Black-box uncertainty quantification shifts the paradigm from inspecting internal states to observing external behavior. By leveraging semantic consistency across multiple samples, adapting conformal prediction to text frequencies, and strategically probing models with follow-up queries or prompt ensembles, researchers can extract highly reliable confidence signals. As demonstrated by frameworks like the one introduced by Hong et al. for clinical outcomes, these strategies allow us to build robust statistical safeguards around the most powerful, opaque models in production today.
Key Foundational Work
Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities (2024) — contributed 2 concept elements to this direction.
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (2024) — contributed 1 concept elements to this direction.
API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access (2024) — contributed 1 concept elements to this direction.