FactCheckmate: Preemptively Detecting and Mitigating Hallucinations in LMs

Deema Alnuhait, Neeraja Kirtane, Muhammad Khalifa, Hao PengSubmission to ICLR 2025 · Withdrawn

TL;DR

Can a language model (LM) predict that it is about to hallucinate before it even generates a single token? FactCheckmate shows that the answer is yes: an LM’s internal hidden states over the input prompt contain strong warning signals about upcoming errors. By attaching lightweight MLPs to intermediate representations, FactCheckmate preemptively detects hallucinations with over 70% accuracy and applies targeted activation steering to improve output factuality by an average of 34.4% with just a 1.2% inference time overhead.

What Problem Does This Paper Address?

Most hallucination mitigation techniques are reactive: they wait until an LM finishes generating text, then use external search tools, verification models, or expensive resampling strategies (like self-consistency) to spot and correct mistakes. These post-hoc pipelines treat the generator as an uninspectable black box, adding significant computational cost and latency.

The authors ask: Can we detect and fix hallucinations before decoding even starts? If an LM’s internal states already reveal uncertainty or false factual associations while encoding the input question, intervention can occur early—saving compute and preventing hallucinations at the source.

Methods

FactCheckmate introduces a two-stage pipeline for preemptive hallucination handling:

Input Prompt ──> LM Layer l Hidden States ──> Pooling ──> Classifier f_θ
                                                              │
                                            Hallucination? (prob < α)
                                                              │
                                                              ▼
Decoded Output <── Next LM Layers <── [ h_I + g_ϕ(h_I) ] <── Steering Model g_ϕ
  1. Preemptive Detection ($f_\theta$):
  2. While the LM processes the input prompt, FactCheckmate extracts the feedforward network (FFN) hidden states from an intermediate layer $l$.
  3. Hidden states are aggregated across input tokens (mean pooling proved most effective).
  4. A lightweight two-layer MLP with a sigmoid head classifies whether the upcoming output will be factual or a hallucination.
  5. Preemptive Mitigation ($g_\phi$):
  6. If $f_\theta$ indicates high risk of hallucinating (confidence threshold $\alpha \le 0.3$), an intervention model $g_\phi$ predicts an additive steering vector.
  7. This vector is added to the last input token's representation $h_I^{(l)}$ before generation begins: $\tilde{h}_I^{(l)} = h_I^{(l)} + g_\phi(h_I^{(l)})$.
  8. The authors evaluate both a deterministic 3-layer MLP and a stochastic variant (predicting Gaussian parameters $\mu$ and $\sigma$ via the reparameterization trick, then sampling candidate adjustments and selecting the one that scores highest under $f_\theta$). Both are trained via Mean Squared Error (MSE) targeting the hidden state produced when the prompt is paired with the ground-truth answer.

Datasets

The authors convert four standard benchmarks into short-answer question-answering (QA) formats to establish ground-truth labels via exact match (EM) and isolate where generation starts:

  • NQ-Open: Open-domain factual QA from Wikipedia.
  • MMLU: Multi-task exam questions spanning STEM and humanities.
  • MedMCQA: Medical entrance examination QA.
  • GSM8K: Multi-step grade school math reasoning.

Key Insights and Ideas

  • Hallucination signals exist before decoding: Models show detectable patterns of factual failure while encoding the prompt. Even when stripping the last 1–3 tokens of the question ($I-n$), detection accuracy remains well above chance.
  • The model states matter, not just question difficulty: When training $f_\theta$ on uncontextualized input word embeddings, accuracy drops to ~50% (random guess). This proves the classifier is detecting internal LM uncertainty rather than simply categorizing "hard" vs. "easy" questions.
  • Middle-to-late layers hold the strongest factuality cues: Classification accuracy peaks in the middle-to-later transformer layers (e.g., layer 14–23 for 7B/8B models), aligning with prior mechanistic interpretability findings that locate factual recall in deep FFN blocks.
  • Negligible disruption to already-correct answers: When checking the false positive rate on MedMCQA, only 9 of 1,362 non-hallucinatory inputs triggered an intervention, and only 2 suffered degraded output quality.

Key Results

  • Preemptive Detection: Evaluated across diverse base and instruction-tuned models (Llama 2, Llama 3/3.1, Mistral, Gemma, and Qwen 2.5; scales from 7B to 70B), the input-only classifier achieves >70% accuracy, often matching or falling within 1–2% of a reactive baseline that inspects both the question and generated response ($I+O$).
  • Factual Improvement: Measured by GPT-4o pairwise evaluation (validated with human annotators, Cohen’s $\kappa = 0.6$), intervened models produced outputs that were 34.4% more factual on average than base model generations. The deterministic intervention achieved a win rate of $\ge 60\%$ across evaluated settings.
  • Inference Efficiency: FactCheckmate adds a modest ~1.2% average decoding time overhead across evaluated 7B/8B models.

Limitations and Future Work

  • Restricted to Short-Answer QA: The study focuses on short-answer tasks where correctness is clearly delimited. Pinpointing the onset of hallucinations in free-form generation (e.g., long-form summarization, multi-turn dialogue) remains an open challenge.
  • Inapplicable to Black-Box APIs: FactCheckmate requires white-box access to extract and modify intermediate hidden states, making it unusable for closed API models (e.g., GPT-4, Claude).
  • Cross-Model Transfer: While classifiers trained on multi-dataset mixtures generalized reasonably well to unseen domains, classifiers struggled to transfer across different model architectures without retraining.
  • Exploration of Other Components: Interventions were applied exclusively to FFN layers; exploring attention heads or key-value caches remains future work.

Summary updated

Abstract

Language models (LMs) hallucinate. We inquire: Can we detect and mitigate hallucinations before they happen? This work answers this research question in the positive, by showing that the internal representations of LMs provide rich signals that can be used for this purpose. We introduce FactCheckMate, which preemptively detects hallucinations by learning a classifier that predicts whether the LM will hallucinate, based on the model's hidden states produced over the inputs, before decoding begins. If a hallucination is detected, FactCheckMate then intervenes, by adjusting the LM's hidden states such that the model will produce more factual outputs. FactCheckMate provides fresh insights that the inner workings of LMs can be revealed by their hidden states. Practically, both the detection and mitigation models in FactCheckMate are lightweight, adding little inference overhead; FactCheckMate proves a more efficient approach for mitigating hallucinations compared to many post-hoc alternatives. We evaluate FactCheckMate over LMs of different scales and model families (including Llama, Mistral, and Gemma), across a variety of QA datasets from different domains. Our results demonstrate the effectiveness of leveraging internal representations for early hallucination detection and mitigation, achieving over 70% preemptive detection accuracy. On average, outputs generated by LMs with intervention are 34.4% more factual compared to those without intervention. The average overhead difference in the inference time introduced by FactCheckMate is around 3.16 seconds.
BibTeX
@misc{alnuhait2024factcheckmate,
  author = {Deema Alnuhait and Neeraja Kirtane and Muhammad Khalifa and Hao Peng},
  title = {{FactCheckmate: Preemptively Detecting and Mitigating Hallucinations in LMs}},
  year = {2024},
  eprint = {2410.02899},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.02899},
}

Loading related content...

Figure