Abstract
Large language models (LLMs) have been shown to possess impressive capabilities, while also raising crucial concerns about the faithfulness of their responses. A primary issue arising in this context is the management of (un)answerable queries by LLMs, which often results in hallucinatory behavior due to overconfidence. In this paper, we explore the behavior of LLMs when presented with (un)answerable queries. We ask: do models represent the fact that the question is (un)answerable when generating a hallucinatory answer? Our results show strong indications that such models encode the answerability of an input query, with the representation of the first decoded token often being a strong indicator. These findings shed new light on the spatial organization within the latent representations of LLMs, unveiling previously unexplored facets of these models. Moreover, they pave the way for the development of improved decoding techniques with better adherence to factual generation, particularly in scenarios where query (un)answerability is a concern.
Introduction
Large Language Models (LLMs) frequently hallucinate answers to unanswerable questions, often driven by a training objective that prioritizes helpfulness. This phenomenon, termed hallucinatory (un)answerability, occurs when a model provides a plausible but false answer to a query that cannot be resolved using the provided context. A fundamental question arises: does the model "know" it is hallucinating? This study investigates whether an LLM's internal hidden states distinguish between answerable and unanswerable queries, even when the generated output is incorrect.
Methodology
The researchers investigated three instruction-tuned models—Flan-T5, Flan-UL2, and OPT-IML—across three benchmarks: SQuAD 2.0, Natural Questions (NQ), and MuSiQue. Four experimental approaches were utilized:
- Prompt Manipulation: Evaluating if explicit "hints" (permission to state a query is unanswerable) reduce hallucinations.
- Beam Relaxation: Analyzing the top $k$ most likely tokens to see if "unanswerable" signals exist below the primary output.
- Linear Probing: Training a logistic regression classifier on the model's last hidden layer at the first generated token to predict answerability.
- Concept Erasure (LEACE): Surgically removing answerability information from hidden states to determine its causal role in model behavior.
Results and Analysis
The study revealed that LLMs encode a generalized concept of answerability linearly within their latent space.
The Answerability Subspace
Hidden states at the first generated token are highly predictive of answerability. Visualizations using Principal Component Analysis (PCA) show distinct clusters for answerable and unanswerable queries. Notably, "hallucinated" responses (unidentified unanswerable queries) consistently occupy a transitional space between these clusters, indicating internal model uncertainty.
Key Findings
- Prompting Efficacy: Adding a simple hint improved unanswerable question detection by up to 80 F1 points in zero-shot settings.
- Linear Separability: Linear probes achieved over 75% F1 in identifying unanswerable queries, even when the model's overt output was a hallucination.
- Causal Evidence: Erasing the "answerability subspace" significantly degraded the model's performance (e.g., SQuAD F1 dropped from 50.1 to 31.2), proving the model relies on these internal signals.
- Cross-Dataset Generalization: Probes trained on SQuAD successfully transferred to NQ, suggesting a dataset-agnostic internal representation of unanswerability.
Conclusion
This research demonstrates that LLMs possess an internal "awareness" of query answerability that is often suppressed by decoding pressures. The existence of a linear answerability subspace suggests that hallucinations are not a result of total ignorance but rather a failure to prioritize internal uncertainty signals. Future work should focus on developing decoding algorithms that leverage these hidden states to trigger "I don't know" responses, enhancing the reliability and factuality of AI systems.
Summary updated