If you ask a large language model (LLM) a difficult question multiple times with a high generation temperature, it will likely provide a variety of different answers. If the model wanders across many distinct responses, we can reasonably infer that it is uncertain. Conversely, if the model repeatedly returns the exact same answer across dozens of independent samples, we tend to assume it "knows" the fact.
This intuition forms the foundation of modern uncertainty estimation and decoding strategies, such as Self-Consistency and Semantic Entropy. The core assumption is simple: variance implies uncertainty, while consensus implies truth.
However, a growing body of research reveals a fundamental flaw in this assumption. LLMs frequently suffer from consistent hallucinations—instances where a model repeatedly, stably, and confidently generates the exact same incorrect answer. When a model's internal state is systematically biased toward a specific error, traditional consensus metrics break down entirely. The model is not guessing; it is unanimously agreeing on a falsehood.
The Illusion of Consensus
To understand why consensus fails, we must look at how uncertainty is typically measured. In Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation, Lorenz Kuhn et al. introduced Semantic Entropy to group syntactically diverse but semantically identical outputs. While this successfully handles linguistic variations, the authors explicitly noted a critical vulnerability: if a model repeatedly generates the same hallucinated fact across all samples, the resulting low entropy score will falsely indicate high confidence in a wrong answer.
This phenomenon represents a complete decoupling of consistency and accuracy. As observed by Ola Shorinwa et al. in their comprehensive survey, LLMs can generate multiple responses that are consistently incorrect, leading to dangerously misleading confidence estimates. Traditional dispersion-based uncertainty methods assume that model errors are accompanied by inconsistent outputs, but as Zhen Lin et al. argue in Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models, if a factual error is deeply ingrained, the response variance drops to zero, making the error completely invisible to statistical consistency checks.
Traditional Assumption:
[Query] ──┬─► Correct Answer A (High Probability) ──► High Consensus = True
├─► Incorrect Answer B (Low Probability)
└─► Incorrect Answer C (Low Probability)
Consistent Hallucination Failure Mode:
[Query] ──┬─► Hallucination X (High Probability) ──► High Consensus = FALSEHOOD
├─► Hallucination X (High Probability)
└─► Hallucination X (High Probability)
Why Models Agree on Lies
Why do language models converge so reliably on incorrect answers? The literature points to three primary mechanisms:
1. Ingrained Training Biases and Token Frequency
The most straightforward reason a model consistently hallucinates is that the error was memorized during training. Furui Cheng et al. demonstrate in RELIC: Investigating Large Language Model Responses using Self-Consistency that self-consistency metrics fail when a model relies on biased or outdated training data. Furthermore, Anders Søgaard et al. show that models are heavily biased toward high-frequency tokens. This bias causes them to stably output common linguistic patterns or frequent facts that are contextually wrong but statistically dominant, masking a total lack of underlying factual knowledge.
2. Strong Internal Priors and Prompt Triggers
Sometimes, the structure of a prompt or a strong internal prior forces the model down a specific logical path. In SAC³: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency, Jiaxin Zhang et al. note that specific prompt triggers can reliably activate systematic hallucinations. Similarly, Zehui Liao et al. show that vision-language models often overrely on strong linguistic priors rather than actual visual evidence, causing them to repeatedly generate identical hallucinations even when presented with contradictory visual data.
3. Flawed Logical Symmetry
Perhaps the most insidious cause is when a model generates a highly coherent but entirely fictional web of logic. For example, in code generation, Bei Chen et al. discovered that when a problem description is ambiguous, a model will consistently generate both functionally incorrect code and matching flawed unit tests. Because the code satisfies the model's own flawed tests, it achieves a deceptively high consensus score.
In another striking example, Yakir Yehuda et al. developed a method called InterrogateLLM, which checks if a model can reconstruct a query from its own answer. They found that when a model is deeply confident in a hallucination, it exhibits a perfect "symmetry of error"—it can flawlessly reconstruct the original question from a completely fabricated answer, bypassing backward-consistency checks.
The Cascading Failures in LLM Systems
The existence of consistent hallucinations creates a dangerous blind spot for engineering frameworks that rely on LLMs. When an error is stable, it propagates unchecked through several common architectural patterns:
- Verification and Post-Editing Triggers: Frameworks like the one proposed by Ruochen Zhao et al. rely on disagreement between reasoning paths to trigger external knowledge retrieval. When a model confidently hallucinates, the lack of variance tricks the system into bypassing the correction mechanisms entirely.
- Early-Stopping and Efficiency Scaling: To save on API costs, frameworks like Early-Stopping Self-Consistency (Yiwei Li et al.) and TrimR (Weizhe Lin et al.) halt the generation budget as soon as a small window of samples reaches a consensus. If the model starts with a systematic error, these systems terminate early, locking in the hallucination and preventing the model from ever exploring alternative, correct paths.
- Model Cascades and Orchestration: Routing frameworks often send queries to cheap, small language models (SLMs) first, escalating to a larger model like GPT-4 only if the small models disagree. As Murong Yue et al. and Chenyu Wang et al. point out, if a weaker model is overconfident in a hallucination due to persistent internal biases, the orchestration layer will falsely assume the answer is reliable and neglect to escalate the task.
- Self-Training Loops: Unsupervised alignment techniques, such as the Entropy Minimized Policy Optimization framework explored by Qingyang Zhang et al., reward models for achieving internal consistency on unlabeled data. If a model is rewarded for stability alone, it will inevitably reinforce its own incorrect but stable reasoning patterns, effectively training itself to become more stubbornly wrong.
Breaking the Mirror
To detect and mitigate consistent hallucinations, researchers are shifting away from simple surface-level output agreement and looking deeper into the model's mechanics.
One approach is to move from black-box text analysis to white-box internal state evaluation. Junqi Jiang et al. introduced Representation Consistency, which measures the alignment of hidden layer activations across reasoning paths rather than just final tokens. However, even internal activations can be deceptively stable. To combat this, Changqing Zhang et al. proposed Semantic Energy, a framework that bypasses normalized probabilities (which can artificially inflate confidence scores) and instead uses raw, unnormalized model logits mapped via a Boltzmann-inspired energy distribution to expose the model's true underlying uncertainty.
Ultimately, the field is realizing that consistency is a measure of a model's conviction, not its accuracy. To build truly reliable systems, AI architectures must treat consensus as a signal of stability, while looking to external verification, structural task decomposition, and raw logit energy to validate the truth.