When a Large Language Model (LLM) hallucinates, we often assume it simply lacks the necessary information. However, a growing body of research suggests a more provocative reality: the model often "knows" the truth internally but fails to express it in its output. This discrepancy—a form of cognitive dissonance—reveals that the generative process itself acts as a bottleneck, filtering or distorting the rich latent knowledge stored within the model's hidden layers.
The Knowing-Saying Gap
The fundamental problem identified across these works is the disconnect between cognitive capability (what is represented in hidden states) and expressive capability (what is selected as the next token). As Yuzi Yan and colleagues argue in Exploring the LLM Journey from Cognition to Expression with Linear Representations, pretraining builds a "brain" capable of internalizing facts, but instruction tuning and RLHF primarily teach that brain how to "speak" in a way that satisfies human-centric formats.
This gap is most visible in tasks where the model must follow rigid constraints. Xiang Chen and Xiaojun Wan found that even when a model's hidden states accurately track whether a constraint has been met, the model often fails to use that information to influence token selection. Similarly, Timothee Mickus et al. observed that models maintain highly accurate internal representations of numeric values even while outputting incorrect arithmetic results, suggesting the failure lies in the transformation or retrieval process rather than a lack of understanding.
The Generative Bottleneck
Why does this information get lost? Researchers point to the final "readout" or decoding stage as a primary culprit. Jörg Schlötterer and Paul E. Youssef suggest that the standard output layer may be unable to resolve ambiguities that the internal vector space has already clarified.
In The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?, Qinyu Zhao and Ming Xu demonstrate that the logit distribution of the very first token often contains "hidden knowledge" about whether a prompt is unanswerable or malicious—knowledge that the subsequent generated sentence might completely ignore. This suggests that the model's internal state is often more honest and better calibrated than its surface-level text.
Probing for Latent Truth
To bridge this gap, researchers use probes—simple classifiers trained on the model's internal activations—to extract the "truth" directly from the hidden layers. Kevin Liu, Stephen Casper, and Dylan Hadfield-Menell found that these internal probes are often better calibrated to uncertainty than the model's text output.
By identifying specific "truth directions" or "truth neurons," we can detect when a model is about to hallucinate. Hadas Orgad and colleagues show in LLMs Know More Than They Show that by probing the hidden states of specific tokens, they can predict the type of mistake a model is about to make before it even happens. Haohang Li further localized this property to specific Truth Neurons that encode truthfulness across different subjects.
Steering Toward Honesty
If the truth is present but ignored, can we force the model to "listen" to its own internal knowledge? This is the goal of Inference-Time Intervention (ITI) and Activation Steering.
- Inference-Time Intervention: Kenneth Li developed ITI to "nudge" internal activations during generation. By shifting activations along a "truthful" direction identified by probes, the model is encouraged to favor its internal factual knowledge over statistical shortcuts.
- Multi-Scale Intervention: Samm Sun et al. introduced Truth Forest, which uses multi-dimensional orthogonal probes to uncover hidden truth without fine-tuning.
- Depth-Wise Steering: Marysia Winkels and Steven Basart found that applying these interventions across the network's depth using a Gaussian distribution—rather than at a single layer—yields more robust and honest reporting.
From Performance to Competence
This research direction shifts our understanding of LLMs from being mere "stochastic parrots" to being "neurolinguistic subjects" with a distinction between competence and performance. Linyang He and Hinrich Schütze argue that LLMs master grammatical and conceptual structures internally long before they can reliably manifest them in text.
Ultimately, as Siheng Li notes in A Survey on the Honesty of Large Language Models, building trustworthy AI requires more than just better data; it requires aligning a model's self-expression with its self-knowledge. By looking "under the hood" at internal representations, we can bypass the limitations of the generative interface and elicit the latent truth that models already possess.