PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics

Derui Zhu, Dingfan Chen, Qing Li, Zongxiong Chen, Lei Ma, Jens Grossklags, Mario Fritz

Abstract

Large Language Models (LLMs) frequently generate factually incorrect "hallucinations." Existing detection methods often rely on external knowledge bases or output probabilities, which can be computationally expensive or insufficient. We present PoLLMgraph, a "white-box" approach that analyzes internal state transition dynamics to detect hallucinations. By modeling hidden layer activations as probabilistic traces through abstract states using Hidden Markov Models (HMMs), PoLLMgraph identifies erratic internal transitions associated with fabrication. Our results demonstrate that PoLLMgraph outperforms existing methods by over 20% in AUC-ROC while requiring minimal training data (fewer than 100 samples).

Introduction

Despite their remarkable capabilities, Large Language Models (LLMs) suffer from "factual hallucinations"—generating statements that sound plausible but are factually incorrect. Current solutions generally fall into two categories: black-box methods, which check output text for inconsistencies against external knowledge bases, and gray-box methods, which analyze output probability or confidence scores.

However, these approaches are often insufficient. Hallucinations are frequently rooted deep within the model's internal representation learning process. Furthermore, black-box methods are computationally expensive and difficult to maintain due to their reliance on massive external databases. PoLLMgraph addresses these issues by looking "under the hood" to catch hallucinations as they occur, utilizing the model's own internal signals.

Methodology: State Transition Dynamics

PoLLMgraph treats the LLM's generation process as a trajectory through a high-dimensional state space. The framework operates through four primary stages:

  1. State Abstraction: To manage the massive dimensionality of LLM hidden layers, the authors employ Principal Component Analysis (PCA) for dimensionality reduction. These reduced vectors are then clustered into "abstract states" using Gaussian Mixture Models (GMM) or simple grid clustering.
  2. Trace Analysis: As the LLM generates tokens, it produces a "trace" of these abstract states.
  3. Probabilistic Modeling: Markov Models (MM) and Hidden Markov Models (HMM) are applied to these traces to learn the typical "rhythms" of factual generation versus the erratic transitions characteristic of hallucinations.
  4. Semantics Binding: Through training on a small set of labeled examples, the HMM learns to associate specific internal state paths with the probability of a factual error.
PoLLMgraph framework overview showing the transition from LLM output to abstract and hidden states for hallucination detection.

Experimental Setup

The researchers evaluated PoLLMgraph across several popular open-source models, including Llama-13B, Alpaca-13B, Vicuna-13B, Llama2-13B, and T5-11B. Evaluation was conducted using two primary benchmarks:

  • TruthfulQA: 817 questions designed to mimic human falsehoods and misconceptions.
  • HaluEval: A large-scale collection of 10,000 QA pairs containing both truthful and hallucinated answers.

Key Results and Insights

PoLLMgraph demonstrates significant improvements over state-of-the-art methods like ITI and SelfCheck, showing an improvement of over 20% in AUC-ROC on the TruthfulQA dataset.

Data Efficiency and Model Agnosticism

A critical finding is the extreme data efficiency of the approach. PoLLMgraph achieves high performance with fewer than 100 training samples, making it highly practical for real-world deployment. Furthermore, the approach is model-agnostic, working consistently across different architectures, including both decoder-only (Llama) and encoder-decoder (T5) models.

Hallucination detection performance (AUCROC) across different LLM architectures as a function of reference data volume.

Interpretability and Robustness

Because PoLLMgraph assigns a hallucination probability to each state transition, it provides token-level interpretability. It can highlight specific words (e.g., a date or name) that triggered the detection. Additionally, the model remains effective even when tested on topics entirely different from its training set, suggesting that the internal "signal" of a hallucination is somewhat universal across subjects.

Discussion and Future Work

While PoLLMgraph is highly effective, its performance is sensitive to hyperparameters such as the number of abstract states and PCA dimensions. Additionally, while it handles moderate topic changes well, significant model fine-tuning or extreme semantic shifts may require retraining the probabilistic models.

Currently, PoLLMgraph serves as a "smoke detector" for hallucinations. Future research will focus on "firefighting"—using these internal state insights to intervene during the generation process and steer the model back toward factual accuracy.

Summary updated

BibTeX
@inproceedings{zhu2024pollmgraph,
  author = {Derui Zhu and Dingfan Chen and Qing Li and Zongxiong Chen and Lei Ma and Jens Grossklags and Mario Fritz},
  title = {{PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.294},
}
Figure