LLM Factoscope: Uncovering LLMs' Factual Discernment through Measuring Inner States

Jinwen He, Yujia Gong, Zijin Lin, Cheng'an Wei, Yue Zhao, Kai ChenACL 2024 · OpenReview

Abstract

Large Language Models (LLMs) frequently generate confident but factually incorrect "hallucinations," posing significant risks in high-stakes domains. We present LLM Factoscope, a novel diagnostic pipeline that detects hallucinations by monitoring a model's internal "physiological" states rather than relying on external database cross-referencing. By analyzing static neuron activations and the dynamic evolution of token probabilities across layers, Factoscope identifies inherent patterns of factual discernment. Our method achieves over 96% accuracy across various architectures, demonstrating that LLMs possess internal signals of truthfulness that can be tapped to improve AI reliability.

Introduction

The primary challenge in deploying LLMs for medicine, law, or education is their tendency to hallucinate. Current mitigation strategies often involve "external verification"—querying massive databases like Wikipedia. However, this approach is computationally expensive and requires constant maintenance of an external truth source. LLM Factoscope explores a different hypothesis: Does the LLM "know" it is lying? Inspired by human polygraphs, this research measures internal states (neuron firing and logit dynamics) to determine if a model is being factual or not.

Methodology

The LLM Factoscope pipeline extracts and analyzes four distinct types of internal features during text generation:

  1. Activation Maps (Static): Identification of "factual-related" neurons that fire with higher intensity during correct information retrieval.
  2. Final Output Ranks (Dynamic): Utilizing the "logit lens" technique to track the rank of the final chosen word as it progresses through the model's layers.
  3. Top-k Output Indices: Analyzing the semantic similarity of candidate words across layers.
  4. Top-k Output Probabilities: Measuring the confidence levels of top candidate words.
Internal state comparison: Activation maps and token rank evolution for factual vs. non-factual outputs.

To process these features, the system employs a Siamese network-based architecture. It uses a ResNet18 CNN to analyze spatial patterns in activations and a Gated Recurrent Unit (GRU) for sequential rank data. The model is trained using Triplet Margin Loss, which mathematically clusters factual internal states together while pushing non-factual states into a separate embedding space.

Key Insights into Factual Discernment

Research reveals that factual discernment is not immediate; it primarily occurs in the middle to later layers (typically after layer 15). In early layers, the model filters for general semantic coherence. As processing continues, factual outputs exhibit high stability: the rank of the correct word rises steadily and remains consistent. In contrast, hallucinations are characterized by wild fluctuations in ranks and probabilities in the final layers, signaling internal uncertainty.

Evolution of top-k candidate tokens and their probabilities across the final layers.

Results and Evaluation

Factoscope was evaluated using a custom dataset of 247,246 data points across categories such as Science, History, and Geography. Testing across architectures including GPT2-XL, Llama2 (7B/13B), and Vicuna yielded the following results:

  • High Accuracy: The system consistently achieved between 96.1% and 98.3% accuracy in hallucination detection.
  • Generalization: A Factoscope trained on history and science data successfully detected hallucinations in unrelated fields like sports, indicating a universal internal signal for factuality.
  • Performance: It significantly outperformed existing white-box (SAPLMA) and black-box (SelfCheckGPT) baselines, particularly as model complexity increased.

Conclusion

LLM Factoscope demonstrates that LLMs possess an internal "factual discernment" that can be measured without external tools. While the method assumes the model's training data was predominantly factual, it opens new avenues for real-time model editing and the development of more trustworthy AI systems. Future work will focus on refining these detectors for highly technical, out-of-distribution jargon.

Summary updated

Abstract

Large Language Models (LLMs) have revolutionized various domains with extensive knowledge and creative capabilities. However, a critical issue with LLMs is their tendency to produce outputs that diverge from factual reality. This phenomenon is particularly concerning in sensitive applications such as medical consultation and legal advice, where accuracy is paramount. Inspired by human lie detectors using physiological responses, we introduce the LLM Factoscope, a novel Siamese network-based model that leverages the inner states of LLMs for factual detection. Our investigation reveals distinguishable patterns in LLMs’ inner states when generating factual versus non-factual content. We demonstrate its effectiveness across various architectures, achieving over 96% accuracy on our custom-collected factual detection dataset. Our work opens a new avenue for utilizing LLMs’ inner states for factual detection and encourages further exploration into LLMs’ inner workings for enhanced reliability and transparency.
BibTeX
@inproceedings{he2024llm,
  author = {Jinwen He and Yujia Gong and Zijin Lin and Cheng'an Wei and Yue Zhao and Kai Chen},
  title = {{LLM Factoscope: Uncovering LLMs' Factual Discernment through Measuring Inner States}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=0FdmuwxjMG},
}
Figure