Abstract
The reliability of large language models (LLMs) is greatly compromised by their tendency to hallucinate, underscoring the need for precise identification of knowledge gaps within LLMs. Various methods for probing such gaps exist, ranging from calibration-based to prompting-based methods. To evaluate these probing methods, in this paper, we propose a new process based on using input variations and quantitative metrics. Through this, we expose two dimensions of inconsistency in knowledge gap probing. (1) Intra-method inconsistency: Minimal non-semantic perturbations in prompts lead to considerable variance in detected knowledge gaps within the same probing method; e.g., the simple variation of shuffling answer options can decrease agreement to around 40%. (2) Cross-method inconsistency: Probing methods contradict each other on whether a model knows the answer. Methods are highly inconsistent – with decision consistency across methods being as low as 7% – even though the model, dataset, and prompt are all the same. These findings challenge existing probing methods and highlight the urgent need for perturbation-robust probing frameworks.
Introduction
As Large Language Models (LLMs) are increasingly integrated into critical decision-making workflows, the risk of "hallucinations"—confident but factually incorrect statements—remains a significant barrier to trust. To address this, researchers employ knowledge probing methods to identify "knowledge gaps," where a model is likely guessing or lacks information. However, the reliability of these diagnostic tools themselves has remained largely unexamined. This study systematically evaluates the consistency of current probing techniques to determine if their assessments are stable and trustworthy.
Methodology
We propose a framework to evaluate probing consistency across two primary dimensions:
- Intra-method consistency: Assessing whether a single probing method yields the same "knows vs. doesn't know" decision when the input prompt undergoes minimal, non-semantic changes, such as shuffling multiple-choice options or introducing minor typos.
- Cross-method consistency: Evaluating whether different probing methods agree on whether a model "knows" a specific fact when given identical input.
We evaluated six categories of probing methods:
- Calibration-based:
TOKPROB(output probabilities) andASKCAL(direct confidence scores). - Training-based:
EMBEDDING(classifiers on internal hidden states). - Prompting-based:
SELFREF(self-reflection) andMOREINFO(asking if more info is needed). - Consistency-based:
NOTA(adding a "None-of-the-Above" option).
Metrics used include Harmonic Consistency IoU ($IoU_{cons}$) for intra-method stability and Decision Consistency ($DecCons$) for cross-method agreement. Evaluations were conducted on the MMLU and HellaSwag benchmarks using models like Mistral-7B and the LLaMA-3 family.
Key Insights and Results
The Reliability Crisis in Probing
Our findings reveal a "reliability crisis" in knowledge gap detection. One of the most striking results is the sensitivity to option shuffling: simply changing the order of multiple-choice answers can cause a probe's agreement with its own previous assessment to drop to as low as 40%. Furthermore, cross-method consistency is alarmingly low, with different probes agreeing on knowledge gaps only 7% of the time in certain configurations.
The Scaling Paradox
Contrary to the expectation that larger models would be more robust, scaling does not consistently improve probing reliability. For instance, the NOTA method's consistency peaked at 3B parameters but declined at 70B. This suggests that larger models may develop more complex, yet more fragile, internal representations that current probes fail to capture consistently.
Impact of Prompting and Calibration
While few-shot prompting typically improves model performance, it can severely degrade probing consistency. For the MOREINFO method, $IoU_{cons}$ plummeted to near zero (0.04) in one-shot settings, as models began to mimic the provided examples rather than performing self-assessment. Additionally, calibration-based methods showed high sensitivity to threshold selection, though this could be mitigated by implementing "threshold safeguards."
Conclusion
Current knowledge probing methods are highly sensitive to surface-level prompt features and often provide contradictory assessments. This instability indicates that "knowing" is not a binary state easily captured by existing tools. There is an urgent need for the development of perturbation-robust probing frameworks that rely on deeper semantic understanding rather than fragile output patterns.
Summary updated