Large Language Models (LLMs) are notorious for hallucinating with confidence. While they can generate fluent, authoritative-sounding prose, their internal probability distributions for long sequences often fail to distinguish between fact and fiction. However, a growing body of research suggests that LLMs possess a "hidden" sense of their own correctness that can be unlocked by changing how we ask them to evaluate themselves.
Instead of looking at the average probability of a generated sentence, researchers have found that forcing a model to perform a binary "Yes/No" verification task provides a much more accurate signal of its internal certainty.
The Failure of Sequence Probability
When an LLM generates a sentence, the probability of that sentence is the product of the probabilities of every individual token. This metric is often a poor proxy for truth because it is heavily biased by linguistic frequency and surface-level fluency. A model might assign a high probability to a common misconception simply because the phrasing is familiar.
Jie Ren and Balaji Lakshminarayanan demonstrated in Self-evaluation Improves Selective Generation in Large Language Models that while LLMs struggle to judge their own quality using standard metrics, they excel when the task is framed as a multiple-choice "exam." By prompting a model to judge its own answer with a simple "Yes" or "No," the probability of the "Yes" token becomes a highly calibrated confidence score. This allows the model to "abstain" from answering when the affirmative token's probability is low, significantly reducing errors in selective generation.
Extracting the Affirmative Signal
The core mechanism of this research direction is the binary verification score. The process typically follows three steps:
- Generation: The model produces a candidate answer or reasoning path.
- Verification Prompting: The model is prompted with its own answer and asked a follow-up question, such as "Is the above answer correct? Answer Yes or No."
- Probability Extraction: Rather than looking at the text the model generates next, researchers extract the softmax probability (or log-probability) assigned to the specific token for "Yes" or "True."
This approach treats self-verification as a classification task. Mark Steyvers and colleagues in What large language models know and what people think they know argue that this secondary inference step generates a probability score that reflects internal certainty more accurately than token likelihoods. Similarly, Boyang Xue showed in MlingConf that this method holds across multiple languages, provided the prompt is aligned with the model's "native" cultural context.
Internal Proxies for Correctness
Beyond explicit "Yes/No" prompts, models appear to have even more direct internal signals of success. Hyunseok Lee and Jihoon Tack introduced ReVISE (ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification), which uses the probability of the termination token (EOS) as a proxy for confidence. The intuition is that a model is more likely to "want" to end a sequence if it "believes" the preceding reasoning is complete and correct.
This concept is taken further in LaSeR (LaSeR: Reinforcement Learning with Last-Token Self-Rewarding) by Wenkai Yang. By aligning the log-probability of a single terminal token with actual correctness, models can be trained via reinforcement learning to internalize verification. This eliminates the need for expensive external reward models, as the model provides its own scalar signal for solution quality at the cost of a single token.
Applications in Reasoning and Consistency
This internal confidence signal is being used to make existing LLM techniques more efficient and accurate:
- Weighted Self-Consistency: Standard self-consistency involves sampling multiple answers and taking a majority vote. Amir Taubenfeld introduced Confidence-Informed Self-Consistency (CISC) (Confidence Improves Self-Consistency in LLMs), which weights these votes by the model's binary verification score. This achieves higher accuracy with 40% fewer samples.
- Fact-and-Reflection: Xinran Zhao and Hongming Zhang developed FaR (Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models), which forces the model to elicit facts before reasoning. The probability of an affirmative response to a follow-up question then serves as a calibration signal to catch hallucinations.
- Bayesian Verification: Dongyoung Kim and Jaehyung Kim proposed ReFeri (Training-free LLM Verification via Recycling Few-shot Examples), which decomposes confidence into "forward" likelihood and "backward" consistency using Bayes' rule, allowing models to verify answers by checking if the answer could logically lead back to the original prompt's examples.
Teaching Models to Know What They Don't Know
While these signals exist intrinsically, they can be significantly improved through targeted training. Sanyam Kapoor and Andrew J. Wilson introduced Calibration-Tuning (CT) (Calibration-Tuning: Teaching Large Language Models to Know What They Don’t Know), a fine-tuning protocol that explicitly teaches models to estimate the probability that their own answers are correct. By training the model to respond to an "uncertainty query," the researchers produced models that are far better at recognizing their own limits.
Despite these advances, challenges remain. Juyeon Heo and Miao Xiong found that while these methods work well for factual knowledge, they still struggle to detect when a model has failed to follow complex, multi-constraint instructions (Do LLMs estimate uncertainty well in instruction-following?). The future of self-aware AI likely lies in deeper probing of these internal states to bridge the gap between what a model says and what it "knows."