Abstract
Language models (LMs) hallucinate. We inquire: Can we detect and mitigate hallucinations before they happen? This work answers this research question in the positive, by showing that the internal representations of LMs provide rich signals that can be used for this purpose. We introduce FactCheckMate, which preemptively detects hallucinations by learning a classifier that predicts whether the LM will hallucinate, based on the model's hidden states produced over the inputs, before decoding begins. If a hallucination is detected, FactCheckMate then intervenes, by adjusting the LM's hidden states such that the model will produce more factual outputs. FactCheckMate provides fresh insights that the inner workings of LMs can be revealed by their hidden states. Practically, both the detection and mitigation models in FactCheckMate are lightweight, adding little inference overhead; FactCheckMate proves a more efficient approach for mitigating hallucinations compared to many post-hoc alternatives. We evaluate FactCheckMate over LMs of different scales and model families (including Llama, Mistral, and Gemma), across a variety of QA datasets from different domains. Our results demonstrate the effectiveness of leveraging internal representations for early hallucination detection and mitigation, achieving over 70% preemptive detection accuracy. On average, outputs generated by LMs with intervention are 34.4% more factual compared to those without intervention. The average overhead difference in the inference time introduced by FactCheckMate is around 3.16 seconds.
Introduction
Large Language Models (LLMs) frequently "hallucinate," generating plausible-sounding but factually incorrect information. Existing solutions are primarily reactive, relying on post-hoc verification via external tools or "judge" models. This approach is computationally expensive and treats the LLM as a "black box," ignoring internal signals that might indicate impending failure. FactCheckmate addresses this by asking: Can we detect and mitigate a hallucination before it is even generated?
Methodology: Preemptive Detection and Intervention
FactCheckmate introduces a two-stage pipeline designed to operate during the prompt processing phase, before decoding begins.
Preemptive Detection ($f_\theta$)
The system identifies "warning signals" within the hidden states produced as the model reads the input prompt. A lightweight binary classifier—a simple two-layer MLP—is trained on pooled hidden states from a specific transformer layer (typically a middle-to-late layer). This classifier predicts the likelihood that the subsequent generation will be a hallucination. Research indicates that detection accuracy peaks in these layers, suggesting they are critical for parametric knowledge retrieval.
Preemptive Mitigation ($g_\phi$)
When a high risk of hallucination is detected, an intervention model adjusts the internal state. It learns a "steering vector" added to the hidden state of the final prompt token, nudging the representation toward a state associated with factual correctness. The framework supports both deterministic (MLP-based) and stochastic (Gaussian-based) intervention models. The stochastic approach allows for sampling multiple potential corrections, selecting the one most trusted by the detector.
Experimental Setup
The framework was evaluated across four diverse Question Answering (QA) domains:
- NQ-open: General knowledge (Wikipedia).
- MMLU: Academic subjects across STEM and humanities.
- MedMCQA: Medical entrance examination questions.
- GSM8K: Grade-school mathematics.
Testing spanned multiple model families, including Llama (2, 3, and 3.1), Mistral, Qwen, and Gemma, with parameter counts ranging from 7B to 70B.
Results and Discussion
FactCheckmate demonstrates that an LLM's internal reaction to a question is a robust predictor of its answer quality.
Performance Metrics
- Detection Accuracy: The preemptive classifier achieved over 70% accuracy across various models, significantly outperforming a random baseline.
- Factuality Improvement: Using GPT-4o as an evaluator, intervened outputs were found to be 34.4% more factual on average than base model outputs.
- Win Rates: In head-to-head comparisons on the NQ-open dataset, the intervened model produced superior answers in over 60% of cases.
- Efficiency: FactCheckmate is highly efficient, adding only approximately 1.2% to total decoding time (an average overhead of 3.16 seconds), making it suitable for real-time applications.
Generalization and Precision
The system exhibits strong generalization; classifiers trained on one model (e.g., Llama-2) perform well on others, suggesting universal structural patterns in how LLMs encode uncertainty. Furthermore, the system maintains a low false positive rate, ensuring minimal interference with questions the model can already answer correctly.
Limitations and Future Work
Despite its success, FactCheckmate faces several challenges. Currently, it is optimized for short-answer QA where correctness is clearly defined; extending this to long-form creative writing or complex multi-step reasoning is a key area for future research. Additionally, the "white-box" nature of the approach requires access to internal activations, precluding its use with closed-source APIs like GPT-4. Future work will also focus on pinpointing the exact tokens that trigger hallucinations in longer sequences.
Conclusion
FactCheckmate demonstrates that LLMs possess internal markers of their own impending failures. By leveraging these signals for preemptive steering, we can significantly enhance the reliability and truthfulness of language model outputs with minimal computational cost.
Summary updated