TL;DR
Not all hallucinations stem from the same root cause: large language models sometimes hallucinate because they genuinely lack knowledge, but other times they err despite possessing the correct answer in their parameters. This paper introduces WACK (Wrong Answer despite having Correct Knowledge), a framework for constructing model-specific datasets to isolate these two failure modes, showing that internal representations differentiate them and that model-specific probing enables preemptive hallucination detection before output generation even begins.
What Problem Does This Paper Address?
Hallucination research in closed-book question answering (CBQA) typically treats factual errors as a monolithic category: an output is either correct or a hallucination. Consequently, detection and mitigation benchmarks rely heavily on "one-size-fits-all" generic datasets.
However, the authors point out a critical distinction:
- $HK^-$ (Hallucination without Knowledge): The model lacks the parametric knowledge required to answer correctly.
- $HK^+$ (Hallucination despite Knowledge): The model stores the correct answer in its weights and can retrieve it under standard prompts, but errs when exposed to prompt noise, misleading context, or persuasion.
Conflating these two causes hinders mitigation. $HK^-$ cannot be resolved internally and requires abstention or retrieval-augmented generation (RAG). Conversely, $HK^+$ errors reside in internal reasoning or decoding failures and can theoretically be corrected via representation engineering or targeted prompting.
Methods
The authors introduce WACK, an automated pipeline to identify and categorize knowledge and failure modes for a specific model:
- Knowledge Filtering: The target model is queried with a prompt across multiple decoding attempts (1 greedy + 5 sampled generations at temperature 0.5). If it consistently fails to produce the gold answer $a_g$, the sample is labeled low-knowledge ($HK^-$). If it consistently outputs $a_g$, it is labeled high-knowledge.
- Inducing Hallucination in High-Knowledge Cases: To force the model to hallucinate on facts it actually knows ($HK^+$), the authors construct two synthetic perturbation prompts:
- Bad-Shots: Prepending few-shot examples with plausible yet false answers to trigger the "snowballing error" effect.
- Alice-Bob: A conversational framing incorporating persuasive context, subtle spelling typos, and low-expectation task constraints.
- Classification & Probing: Linear Support Vector Machines (SVMs) are trained on hidden representations (residual stream, MLP, and attention activations) extracted at:
- The final token of the generated answer (post-hoc detection).
- The final token of the question prior to generating any answer tokens (preemptive detection).
Datasets
The authors use subsets (30,000 short-answer questions each) of two popular open-domain QA benchmarks:
- TriviaQA
- Natural Questions (NQ)
Using these bases, they build model-specific WACK datasets for three 7B–9B open models: Mistral-7B-v0.3, Llama-3.1-8B, and Gemma-2-9B. They also construct a generic baseline dataset by using Mistral to fabricate plausible incorrect answers for the question triplets $(q, a_g, a_h)$.
Key Insights and Ideas
- Distinct Internal Signatures: Probing reveals that the model's inner activations distinctly separate ignorance ($HK^-$) from prompt-induced error ($HK^+$) and factual correctness. The model "knows" internally when it has the fact vs. when it does not.
- Hallucinations Are Model-Specific: Even when models share knowledge of the same facts (Jaccard similarity of 0.6–0.8), the specific prompts that trigger hallucinations differ across model families (Jaccard similarity drops by 0.1–0.2). Generic, shared hallucination datasets miss these model-specific nuances.
- Mitigation Disparity: Simple factual prompting (e.g., instructing the model to remain strictly factual) rescues $HK^+$ hallucinations, but does almost nothing for $HK^-$, proving that these categories require fundamentally different treatment.
- Preemptive Detection is Possible: Because $HK^+$ datasets map inputs directly to a model's propensity to err on known facts, linear probes on the question's final token can predict whether a model is about to hallucinate before the first token of the answer is sampled.
Key Results
- 3-Way State Classification: Probes classifying states into $HK^-$, $HK^+$, and factually correct achieved 60%–70% accuracy across residual layers (compared to a 33% random baseline). Binary pairwise accuracy in the middle layers reached $\ge 70\%$.
- Prompt Mitigation: Adding an instruction to be factual recovered $HK^+$ errors substantially (e.g., from 0% pre-mitigation to 48.7% / 49.4% on Mistral for TriviaQA and NQ), whereas $HK^-$ accuracy remained virtually unchanged (~5%–8%).
- Model-Specific vs. Generic Detectors: Classifiers trained on WACK datasets consistently outperformed those trained on generic datasets when detecting $HK^+$ instances.
- Preemptive Detection: When evaluating hidden states at the prompt’s end token, model-specific probes predicted hallucinations with 60%–80% accuracy across layers. In contrast, probes trained on generic post-hoc datasets collapsed to random chance (~50% or worse).
Limitations and Future Work
- Evaluated Scope: Experiments were confined to three dense models in the 7B–9B parameter range; trends may differ in larger or proprietary mixture-of-experts (MoE) architectures.
- Bimodal Simplification: The study isolates the extremes of the spectrum (strictly known vs. strictly unknown), discarding the middle ground (~5% of cases where the model occasionally answers correctly).
- Induction Methods: The paper focuses on two perturbation styles (bad-shot snowballing and persuasive roleplay). Exploring other natural user errors, complex reasoning chains, or adversarial contexts remains future work.
- Matching Heuristics: Answer evaluation relied on exact-string matching, which might miscategorize answers that are semantically valid paraphrases.
Summary updated