Uncertainty Quantification for Black Box Language Models

This direction is split into other directions: black-box uncertainty estimation. Its last version stays readable and citable here.

Version 1 of 1221 papers247 conceptsfrozen 2026-10-04

When we use a Large Language Model (LLM) like GPT-4, we are interacting with a "black box." We can see the text it produces, but we cannot see its "internal monologue"—the probability distributions (logits) or hidden layer activations that indicate how certain the model is about its own words.

In traditional machine learning, we use these internal signals to measure uncertainty. Without them, we face a dangerous "observability gap": a model might state a hallucination with the same authoritative tone as a verified fact. To bridge this gap, researchers are developing methods to quantify uncertainty using only the model's external behavior.

The Consistency Intuition

The most intuitive way to tell if a model is "guessing" is to see if it can tell the same story twice. If you ask a model a question five times and it gives five different answers, it is likely uncertain.

This is the core idea behind SelfCheckGPT, developed by Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. By sampling multiple stochastic responses to the same prompt, their system identifies factual errors based on how much these samples contradict one another. Similarly, Jiaxin Zhang and colleagues introduced SAC³, which uses "semantic-aware cross-check consistency." Instead of looking for exact word matches, SAC³ checks if the meaning of the answers remains stable across different ways of asking the same question.

Zhen Lin and Jimeng Sun took this further by treating these multiple responses as nodes in a semantic graph. By measuring the "eccentricity" of these nodes—how far one answer drifts from the others—they can mathematically predict when a model is likely to be wrong without ever looking at a logit.

Just Ask for Confidence

If we can't see the model's internal math, why not just ask it how sure it is?

Surprisingly, Eric Mitchell and Chelsea Finn found that while Reinforcement Learning from Human Feedback (RLHF) often ruins a model's internal calibration (making its internal probabilities overconfident), the model's verbalized confidence remains remarkably accurate. Simply asking a model to "state your confidence from 0 to 100" can reduce calibration error by up to 50%.

However, this "self-reflection" isn't always perfect. Makoto Yamada and Yao-Hung Hubert Tsai observed a scaling law in this ability: while small models are hopelessly overconfident, larger models are significantly better at "knowing what they know" when prompted correctly.

Probing and Perturbation

Another strategy is to "stress test" the model's response. If a model's answer changes when you slightly paraphrase the question, it's a sign of "atypicality"—the model is operating in a region of its knowledge space where it is unstable.

Lingyong Yan and Weiwei Sun developed a Self-Detection method that combines this consistency check with a measure of how "unfamiliar" the phrasing of a question is to the model. Similarly, Kyomin Jung and Jahyun Koo found that forcing a model to express uncertainty (e.g., by adding the prefix "I am not sure but...") causes hallucinations to vanish or change, while factual answers remain stable.

Surrogate Models and Statistical Guarantees

When the black box is too opaque, we can use a "spy" to watch it. Percy Liang and Vaishnavi Shrivastava demonstrated that open-source surrogate models (like Llama 2) can be used to estimate the confidence of closed models like GPT-4. Often, the weaker model’s "gut feeling" about a prompt is a better indicator of the stronger model's reliability than the stronger model's own self-reports.

For applications requiring high reliability, researchers are turning to conformal prediction. Inseop Lee introduced TRAQ, a framework that uses multiple stochastic passes to generate a set of potential answers. This set is mathematically guaranteed to contain the correct response with a user-defined level of probability, providing the "error bars" necessary for rigorous science and medicine.

By shifting the focus from internal weights to external behaviors—consistency, verbalization, and stability—this research direction allows us to trust "black box" AI in high-stakes environments where "I think so" isn't a good enough answer.

Papers in this version

Showing 60 of 221 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_uncertainty_quantification_for_black_box_f93f15d2,
  title = {Uncertainty Quantification for Black Box Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uncertainty-quantification-for-black-box-language-models/rdr_f93f15d23f504d3fa9fe8ef717f2cee9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_uncertainty_quantification_for_black_box_f93f15d2_v1,
  title = {Uncertainty Quantification for Black Box Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uncertainty-quantification-for-black-box-language-models/rdr_f93f15d23f504d3fa9fe8ef717f2cee9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}