Distinguishing Ignorance from Error in LLM Hallucinations

Adi Simhi, Jonathan Herzig, Idan Szpektor, Yonatan BelinkovSubmission to ICLR 2025 · Withdrawn

TL;DR

Not all hallucinations stem from the same root cause: large language models sometimes hallucinate because they genuinely lack knowledge, but other times they err despite possessing the correct answer in their parameters. This paper introduces WACK (Wrong Answer despite having Correct Knowledge), a framework for constructing model-specific datasets to isolate these two failure modes, showing that internal representations differentiate them and that model-specific probing enables preemptive hallucination detection before output generation even begins.

What Problem Does This Paper Address?

Hallucination research in closed-book question answering (CBQA) typically treats factual errors as a monolithic category: an output is either correct or a hallucination. Consequently, detection and mitigation benchmarks rely heavily on "one-size-fits-all" generic datasets.

However, the authors point out a critical distinction:

  1. $HK^-$ (Hallucination without Knowledge): The model lacks the parametric knowledge required to answer correctly.
  2. $HK^+$ (Hallucination despite Knowledge): The model stores the correct answer in its weights and can retrieve it under standard prompts, but errs when exposed to prompt noise, misleading context, or persuasion.

Conflating these two causes hinders mitigation. $HK^-$ cannot be resolved internally and requires abstention or retrieval-augmented generation (RAG). Conversely, $HK^+$ errors reside in internal reasoning or decoding failures and can theoretically be corrected via representation engineering or targeted prompting.

Methods

The authors introduce WACK, an automated pipeline to identify and categorize knowledge and failure modes for a specific model:

  1. Knowledge Filtering: The target model is queried with a prompt across multiple decoding attempts (1 greedy + 5 sampled generations at temperature 0.5). If it consistently fails to produce the gold answer $a_g$, the sample is labeled low-knowledge ($HK^-$). If it consistently outputs $a_g$, it is labeled high-knowledge.
  2. Inducing Hallucination in High-Knowledge Cases: To force the model to hallucinate on facts it actually knows ($HK^+$), the authors construct two synthetic perturbation prompts:
  3. Bad-Shots: Prepending few-shot examples with plausible yet false answers to trigger the "snowballing error" effect.
  4. Alice-Bob: A conversational framing incorporating persuasive context, subtle spelling typos, and low-expectation task constraints.
  5. Classification & Probing: Linear Support Vector Machines (SVMs) are trained on hidden representations (residual stream, MLP, and attention activations) extracted at:
  6. The final token of the generated answer (post-hoc detection).
  7. The final token of the question prior to generating any answer tokens (preemptive detection).

Datasets

The authors use subsets (30,000 short-answer questions each) of two popular open-domain QA benchmarks:

  • TriviaQA
  • Natural Questions (NQ)

Using these bases, they build model-specific WACK datasets for three 7B–9B open models: Mistral-7B-v0.3, Llama-3.1-8B, and Gemma-2-9B. They also construct a generic baseline dataset by using Mistral to fabricate plausible incorrect answers for the question triplets $(q, a_g, a_h)$.

Key Insights and Ideas

  • Distinct Internal Signatures: Probing reveals that the model's inner activations distinctly separate ignorance ($HK^-$) from prompt-induced error ($HK^+$) and factual correctness. The model "knows" internally when it has the fact vs. when it does not.
  • Hallucinations Are Model-Specific: Even when models share knowledge of the same facts (Jaccard similarity of 0.6–0.8), the specific prompts that trigger hallucinations differ across model families (Jaccard similarity drops by 0.1–0.2). Generic, shared hallucination datasets miss these model-specific nuances.
  • Mitigation Disparity: Simple factual prompting (e.g., instructing the model to remain strictly factual) rescues $HK^+$ hallucinations, but does almost nothing for $HK^-$, proving that these categories require fundamentally different treatment.
  • Preemptive Detection is Possible: Because $HK^+$ datasets map inputs directly to a model's propensity to err on known facts, linear probes on the question's final token can predict whether a model is about to hallucinate before the first token of the answer is sampled.

Key Results

  • 3-Way State Classification: Probes classifying states into $HK^-$, $HK^+$, and factually correct achieved 60%–70% accuracy across residual layers (compared to a 33% random baseline). Binary pairwise accuracy in the middle layers reached $\ge 70\%$.
  • Prompt Mitigation: Adding an instruction to be factual recovered $HK^+$ errors substantially (e.g., from 0% pre-mitigation to 48.7% / 49.4% on Mistral for TriviaQA and NQ), whereas $HK^-$ accuracy remained virtually unchanged (~5%–8%).
  • Model-Specific vs. Generic Detectors: Classifiers trained on WACK datasets consistently outperformed those trained on generic datasets when detecting $HK^+$ instances.
  • Preemptive Detection: When evaluating hidden states at the prompt’s end token, model-specific probes predicted hallucinations with 60%–80% accuracy across layers. In contrast, probes trained on generic post-hoc datasets collapsed to random chance (~50% or worse).

Limitations and Future Work

  • Evaluated Scope: Experiments were confined to three dense models in the 7B–9B parameter range; trends may differ in larger or proprietary mixture-of-experts (MoE) architectures.
  • Bimodal Simplification: The study isolates the extremes of the spectrum (strictly known vs. strictly unknown), discarding the middle ground (~5% of cases where the model occasionally answers correctly).
  • Induction Methods: The paper focuses on two perturbation styles (bad-shot snowballing and persuasive roleplay). Exploring other natural user errors, complex reasoning chains, or adversarial contexts remains future work.
  • Matching Heuristics: Answer evaluation relied on exact-string matching, which might miscategorize answers that are semantically valid paraphrases.

Summary updated

Abstract

Large language models (LLMs) are susceptible to hallucinations---outputs that are ungrounded, factually incorrect, or inconsistent with prior generations. We focus on close-book Question Answering (CBQA), where previous work has not fully addressed the distinction between two possible kinds of hallucinations, namely, whether the model (1) does not hold the correct answer in its parameters or (2) answers incorrectly despite having the required knowledge. We argue that distinguishing these cases is crucial for detecting and mitigating hallucinations. Specifically, case (2) may be mitigated by intervening in the model’s internal computation, as the knowledge resides within the model's parameters. In contrast, in case (1) there is no parametric knowledge to leverage for mitigation, so it should be addressed by resorting to an external knowledge source or abstaining. To help distinguish between the two cases, we introduce Wrong Answer despite having Correct Knowledge (WACK), an approach for constructing model-specific datasets for the second hallucination type. Our probing experiments indicate that the two kinds of hallucinations are represented differently in the model's inner states. Next, we show that datasets constructed using WACK exhibit variations across models, demonstrating that even when models share knowledge of certain facts, they still vary in the specific examples that lead to hallucinations. Finally, we show that training a probe on our WACK datasets leads to better hallucination detection of case (2) hallucinations than using the common generic one-size-fits-all datasets.
BibTeX
@misc{simhi2024distinguishing,
  author = {Adi Simhi and Jonathan Herzig and Idan Szpektor and Yonatan Belinkov},
  title = {{Distinguishing Ignorance from Error in LLM Hallucinations}},
  year = {2024},
  eprint = {2410.22071},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.22071},
}
Figure