Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probing

Raoyuan Zhao, Abdullatif Köksal, Ali Modarressi, Michael A. Hedderich, Hinrich Schütze

Abstract

The reliability of large language models (LLMs) is greatly compromised by their tendency to hallucinate, underscoring the need for precise identification of knowledge gaps within LLMs. Various methods for probing such gaps exist, ranging from calibration-based to prompting-based methods. To evaluate these probing methods, in this paper, we propose a new process based on using input variations and quantitative metrics. Through this, we expose two dimensions of inconsistency in knowledge gap probing. (1) Intra-method inconsistency: Minimal non-semantic perturbations in prompts lead to considerable variance in detected knowledge gaps within the same probing method; e.g., the simple variation of shuffling answer options can decrease agreement to around 40%. (2) Cross-method inconsistency: Probing methods contradict each other on whether a model knows the answer. Methods are highly inconsistent – with decision consistency across methods being as low as 7% – even though the model, dataset, and prompt are all the same. These findings challenge existing probing methods and highlight the urgent need for perturbation-robust probing frameworks.

Introduction

As Large Language Models (LLMs) are increasingly integrated into critical decision-making workflows, the risk of "hallucinations"—confident but factually incorrect statements—remains a significant barrier to trust. To address this, researchers employ knowledge probing methods to identify "knowledge gaps," where a model is likely guessing or lacks information. However, the reliability of these diagnostic tools themselves has remained largely unexamined. This study systematically evaluates the consistency of current probing techniques to determine if their assessments are stable and trustworthy.

Methodology

We propose a framework to evaluate probing consistency across two primary dimensions:

  1. Intra-method consistency: Assessing whether a single probing method yields the same "knows vs. doesn't know" decision when the input prompt undergoes minimal, non-semantic changes, such as shuffling multiple-choice options or introducing minor typos.
  2. Cross-method consistency: Evaluating whether different probing methods agree on whether a model "knows" a specific fact when given identical input.
Conceptual framework for evaluating Intra-Method and Cross-Method consistency in knowledge probing.

We evaluated six categories of probing methods:

  • Calibration-based: TOKPROB (output probabilities) and ASKCAL (direct confidence scores).
  • Training-based: EMBEDDING (classifiers on internal hidden states).
  • Prompting-based: SELFREF (self-reflection) and MOREINFO (asking if more info is needed).
  • Consistency-based: NOTA (adding a "None-of-the-Above" option).

Metrics used include Harmonic Consistency IoU ($IoU_{cons}$) for intra-method stability and Decision Consistency ($DecCons$) for cross-method agreement. Evaluations were conducted on the MMLU and HellaSwag benchmarks using models like Mistral-7B and the LLaMA-3 family.

Key Insights and Results

The Reliability Crisis in Probing

Our findings reveal a "reliability crisis" in knowledge gap detection. One of the most striking results is the sensitivity to option shuffling: simply changing the order of multiple-choice answers can cause a probe's agreement with its own previous assessment to drop to as low as 40%. Furthermore, cross-method consistency is alarmingly low, with different probes agreeing on knowledge gaps only 7% of the time in certain configurations.

Decision consistency heatmaps across different models and datasets, showing significant disagreement between probing methods.

The Scaling Paradox

Contrary to the expectation that larger models would be more robust, scaling does not consistently improve probing reliability. For instance, the NOTA method's consistency peaked at 3B parameters but declined at 70B. This suggests that larger models may develop more complex, yet more fragile, internal representations that current probes fail to capture consistently.

Impact of Prompting and Calibration

While few-shot prompting typically improves model performance, it can severely degrade probing consistency. For the MOREINFO method, $IoU_{cons}$ plummeted to near zero (0.04) in one-shot settings, as models began to mimic the provided examples rather than performing self-assessment. Additionally, calibration-based methods showed high sensitivity to threshold selection, though this could be mitigated by implementing "threshold safeguards."

Conclusion

Current knowledge probing methods are highly sensitive to surface-level prompt features and often provide contradictory assessments. This instability indicates that "knowing" is not a binary state easily captured by existing tools. There is an urgent need for the development of perturbation-robust probing frameworks that rely on deeper semantic understanding rather than fragile output patterns.

Summary updated

Abstract

The reliability of large language models (LLMs) is greatly compromised by their tendency to hallucinate, underscoring the need for precise identification of knowledge gaps within LLMs. Various methods for probing such gaps exist, ranging from calibration-based to prompting-based methods. To evaluate these probing methods, in this paper, we propose a new process based on using input variations and quantitative metrics. Through this, we expose two dimensions of inconsistency in knowledge gap probing. (1) **Intra-method inconsistency:** Minimal non-semantic perturbations in prompts lead to considerable variance in detected knowledge gaps within the same probing method; e.g., the simple variation of shuffling answer options can decrease agreement to around 40%. (2) **Cross-method inconsistency:** Probing methods contradict each other on whether a model knows the answer. Methods are highly inconsistent – with decision consistency across methods being as low as 7% – even though the model, dataset, and prompt are all the same. These findings challenge existing probing methods and highlight the urgent need for perturbation-robust probing frameworks.
BibTeX
@inproceedings{zhao2025do,
  author = {Raoyuan Zhao and Abdullatif Köksal and Ali Modarressi and Michael A. Hedderich and Hinrich Schütze},
  title = {{Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=58J59uuvxo},
}
Figure