Large Language Models (LLMs) are notorious for their confidence, even when they are completely wrong. This specific type of hallucination—where a model fabricates detailed facts about entities that do not exist—is known as confabulation. While the phenomenon is well-documented, the internal mechanisms driving these fabrications have remained largely opaque.
A recent study from researchers at Freie Universität Berlin, titled "Where Confabulation Lives: Latent Feature Discovery in LLMs," provides a mechanistic look at this behavior. By identifying specific "latent directions" in the model's activation space, the researchers demonstrate that confabulation is a distinct, steerable state rather than a simple failure of memory.
Methodology: Contrastive Probing and SPCA
To isolate the internal representation of confabulation, the researchers employed a three-step approach:
- Contrastive Probing: The model was prompted with pairs of questions—one regarding a well-known entity (e.g., "Who is Nelson Mandela?") and another regarding a fabricated entity (e.g., "Who is Zahir Mansour?").
- Sparse Feature Extraction: Rather than using simple mean differences, the team utilized Sparse Principal Component Analysis (SPCA). This method identifies "disentangled" directions in the activation space that are specifically tied to the act of confabulating while filtering out background linguistic noise.
- Activation Steering: By subtracting the identified "confabulation vector" from the model's internal representations during the forward pass, the researchers could suppress the urge to fabricate without damaging the model's general linguistic capabilities.
The Locus of Confabulation
A critical finding of the study is that the signal for confabulation is not distributed evenly across the model. Instead, it peaks in the middle layers (e.g., Layer 14 for Llama3-8B). These layers appear to represent the model's internal "reasoning" or "knowledge state," situated between low-level token processing and high-level output generation.
Abstract Generalization Across Domains
Perhaps the most significant insight is that confabulation is an abstract behavioral feature. A steering vector derived from data about proteins can successfully detect and suppress confabulations about celebrities or legal advice. This suggests that the model enters a specific "confabulation state" that is independent of the subject matter.
Key Results and Implications
- High Detection Accuracy: The discovered latent directions allowed for classifiers to detect confabulation with over 90% accuracy across most categories.
- Asymmetrical Control: It is easier to suppress the "confabulation" state than it is to break the "factual" state. Steering a model to stop lying about fake entities does not necessarily cause it to forget real ones.
- Sparsity is Crucial: Using sparse vectors (SPCA) was essential for maintaining model coherence. While simple mean-activation steering often led to incoherent output, sparse steering modulated behavior while keeping the language fluent.
Conclusion
This research moves beyond treating LLMs as black boxes and toward a future of token-level control. By identifying the specific internal activations that represent the "intent" to fabricate, we can develop models that are more aware of their own knowledge boundaries. Future work will likely focus on integrating these steering methods into Reinforcement Learning (RLHF) to create inherently more honest AI systems.
Summary updated