The Curious Case of Hallucinatory Unanswerablity: Finding Truths in the Hidden States of Over-Confident Large Language Models

Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, Shauli Ravfogel

Abstract

Large language models (LLMs) have been shown to possess impressive capabilities, while also raising crucial concerns about the faithfulness of their responses. A primary issue arising in this context is the management of (un)answerable queries by LLMs, which often results in hallucinatory behavior due to overconfidence. In this paper, we explore the behavior of LLMs when presented with (un)answerable queries. We ask: do models represent the fact that the question is (un)answerable when generating a hallucinatory answer? Our results show strong indications that such models encode the answerability of an input query, with the representation of the first decoded token often being a strong indicator. These findings shed new light on the spatial organization within the latent representations of LLMs, unveiling previously unexplored facets of these models. Moreover, they pave the way for the development of improved decoding techniques with better adherence to factual generation, particularly in scenarios where query (un)answerability is a concern.

Introduction

Large Language Models (LLMs) frequently hallucinate answers to unanswerable questions, often driven by a training objective that prioritizes helpfulness. This phenomenon, termed hallucinatory (un)answerability, occurs when a model provides a plausible but false answer to a query that cannot be resolved using the provided context. A fundamental question arises: does the model "know" it is hallucinating? This study investigates whether an LLM's internal hidden states distinguish between answerable and unanswerable queries, even when the generated output is incorrect.

Methodology

The researchers investigated three instruction-tuned models—Flan-T5, Flan-UL2, and OPT-IML—across three benchmarks: SQuAD 2.0, Natural Questions (NQ), and MuSiQue. Four experimental approaches were utilized:

  1. Prompt Manipulation: Evaluating if explicit "hints" (permission to state a query is unanswerable) reduce hallucinations.
  2. Beam Relaxation: Analyzing the top $k$ most likely tokens to see if "unanswerable" signals exist below the primary output.
  3. Linear Probing: Training a logistic regression classifier on the model's last hidden layer at the first generated token to predict answerability.
  4. Concept Erasure (LEACE): Surgically removing answerability information from hidden states to determine its causal role in model behavior.
Experimental prompt templates with and without (un)answerability hints.

Results and Analysis

The study revealed that LLMs encode a generalized concept of answerability linearly within their latent space.

The Answerability Subspace

Hidden states at the first generated token are highly predictive of answerability. Visualizations using Principal Component Analysis (PCA) show distinct clusters for answerable and unanswerable queries. Notably, "hallucinated" responses (unidentified unanswerable queries) consistently occupy a transitional space between these clusters, indicating internal model uncertainty.

3D PCA projections of hidden states across SQuAD, NQ, and MuSiQue datasets.

Key Findings

  • Prompting Efficacy: Adding a simple hint improved unanswerable question detection by up to 80 F1 points in zero-shot settings.
  • Linear Separability: Linear probes achieved over 75% F1 in identifying unanswerable queries, even when the model's overt output was a hallucination.
  • Causal Evidence: Erasing the "answerability subspace" significantly degraded the model's performance (e.g., SQuAD F1 dropped from 50.1 to 31.2), proving the model relies on these internal signals.
  • Cross-Dataset Generalization: Probes trained on SQuAD successfully transferred to NQ, suggesting a dataset-agnostic internal representation of unanswerability.

Conclusion

This research demonstrates that LLMs possess an internal "awareness" of query answerability that is often suppressed by decoding pressures. The existence of a linear answerability subspace suggests that hallucinations are not a result of total ignorance but rather a failure to prioritize internal uncertainty signals. Future work should focus on developing decoding algorithms that leverage these hidden states to trigger "I don't know" responses, enhancing the reliability and factuality of AI systems.

Summary updated

Abstract

Large language models (LLMs) have been shown to possess impressive capabilities, while also raising crucial concerns about the faithfulness of their responses. A primary issue arising in this context is the management of (un)answerable queries by LLMs, which often results in hallucinatory behavior due to overconfidence. In this paper, we explore the behavior of LLMs when presented with (un)answerable queries. We ask: do models represent the fact that the question is (un)answerable when generating a hallucinatory answer? Our results show strong indications that such models encode the answerability of an input query, with the representation of the first decoded token often being a strong indicator. These findings shed new light on the spatial organization within the latent representations of LLMs, unveiling previously unexplored facets of these models. Moreover, they pave the way for the development of improved decoding techniques with better adherence to factual generation, particularly in scenarios where query (un)answerability is a concern.
BibTeX
@inproceedings{slobodkin2023curious,
  author = {Aviv Slobodkin and Omer Goldman and Avi Caciularu and Ido Dagan and Shauli Ravfogel},
  title = {{The Curious Case of Hallucinatory Unanswerablity: Finding Truths in the Hidden States of Over-Confident Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=mCnBRLJuhY},
}
Figure