From Confidence to Collapse in LLM Factual Robustness

Alina Fastowski, Bardh Prenkaj, Gjergji Kasneci

Abstract

Ensuring the robustness of factual knowledge in LLMs is critical for reliable applications in tasks such as question answering and reasoning. However, existing evaluation methods predominantly focus on performance-based metrics, often investigating from the perspective of prompt perturbations, which captures only the externally triggered side of knowledge robustness. To bridge this gap, we introduce a principled approach to measure factual robustness from the perspective of the generation process by analyzing token distribution entropy in combination with temperature scaling sensitivity. These two factors build the Factual Robustness Score (FRS), a novel metric which quantifies the stability of a fact against perturbations in decoding conditions, given its initial uncertainty. To validate our approach, we conduct extensive experiments on 5 LLMs across 3 closed-book QA datasets (SQuAD, TriviaQA, and HotpotQA). We show that factual robustness varies significantly - smaller models report an FRS of 0.76, larger ones 0.93 - with accuracy degrading by 60% under increased uncertainty. These insights demonstrate how entropy and temperature scaling impact factual accuracy, and lay a foundation for developing more robust knowledge retention and retrieval in future models.

Introduction

Standard evaluation of Large Language Models (LLMs) typically relies on "snapshot" accuracy: whether a model provides the correct answer under deterministic conditions (temperature $t=0$). However, this metric can be deceptive. A model might correctly identify a fact in a single instance but fail if the decoding conditions are slightly perturbed. This indicates a lack of internal robustness—the reliability with which a fact is stored within the model's weights.

We argue that current evaluations miss this internal dimension. If a model is essentially "guessing" with high uncertainty, its knowledge is fragile. To address this, we propose the Factual Robustness Score (FRS), a metric designed to quantify how "stably" an LLM holds onto its factual knowledge by stress-testing it against increasing sampling temperatures.

Methodology: The Factual Robustness Score

The FRS is built upon two fundamental pillars of the generation process:

  1. Token Distribution Entropy ($H$): This measures intrinsic uncertainty. Low entropy indicates high confidence in a specific token, while high entropy suggests the model is spreading probability across multiple candidates.
  2. Breaking Temperature ($t_b$): We systematically increase the sampling temperature (introducing "noise") until the model's accuracy on a specific fact drops below a 50% threshold. This $t_b$ represents the model's collapse point.
The relationship between initial entropy and the breaking temperature ($t_b$) across different datasets and models.

The FRS integrates these factors, rewarding models that are both confident and resilient. It specifically penalizes "lucky guesses"—instances where the model is correct at $t=0$ but exhibits high internal uncertainty.

Experimental Setup

We evaluated five state-of-the-art LLMs: GPT-4o-mini, LLaMA-3 (3b, 8b), and Qwen-2.5 (3b, 14b). These models were tested across three benchmark closed-book Question Answering (QA) datasets:

  • SQuAD: Wikipedia-based comprehension.
  • TriviaQA: Knowledge-intensive trivia.
  • HotpotQA: Multi-hop reasoning.

Additionally, we categorized questions using a classifier to analyze robustness across different knowledge domains, such as Numerical, Location, and Human entities.

Key Findings

The Performance Collapse

Our experiments reveal a significant gap between initial accuracy and factual robustness. As temperature increases, accuracy across all models and datasets degrades sharply.

Accuracy degradation across SQuAD, TriviaQA, and HotpotQA as sampling temperature $t$ increases.

Smaller models, such as LLaMA-3.2-3B, showed the most dramatic declines, losing over 60% of their accuracy as $t$ approached 2.0. In contrast, larger or more optimized models like Qwen-2.5-14B maintained higher accuracy for longer, indicating more "deeply" embedded knowledge.

Architecture vs. Scale

While model size generally correlates with robustness, architecture and training data are often more decisive. For instance, the Qwen-2.5-3B model outperformed the larger LLaMA-3.1-8B in robustness scores. This suggests that how a model is trained impacts its internal certainty more than raw parameter count.

Domain-Specific Robustness

Numerical facts (dates, quantities) and locations proved to be the most "sticky" or robust across all models. Conversely, "Human" facts (names of people or groups) were the most fragile. We hypothesize that because names often involve longer token sequences, there are more points of failure where the model can "derail" during the generation process under noise.

Conclusion

By shifting the focus from binary accuracy to the FRS, we provide a more nuanced view of LLM reliability. Our results show that high accuracy at $t=0$ does not guarantee factual stability. The FRS offers a vital tool for developers of high-stakes AI applications, where understanding the "breaking point" of a model's knowledge is as important as its initial performance.

Code available at: https://github.com/afastowski/frs

Summary updated

Abstract

Ensuring the robustness of factual knowledge in LLMs is critical for reliable applications in tasks such as question answering and reasoning.However, existing evaluation methods predominantly focus on performance-based metrics, often investigating from the perspective of prompt perturbations, which captures only the externally triggered side of knowledge robustness.To bridge this gap, we introduce a principled approach to measure factual robustness from the perspective of the generation process by analyzing token distribution entropy in combination with temperature scaling sensitivity.These two factors build the Factual Robustness Score (FRS), a novel metric which quantifies the stability of a fact against perturbations in decoding conditions, given its initial uncertainty.To validate our approach, we conduct extensive experiments on 5 LLMs across 3 closed-book QA datasets (SQuAD, TriviaQA, and HotpotQA).We show that factual robustness varies significantly -smaller models report an FRS of 0.76, larger ones 0.93 -with accuracy degrading by 60% under increased uncertainty.These insights demonstrate how entropy and temperature scaling impact factual accuracy, and lay a foundation for developing more robust knowledge retention and retrieval in future models.We release our code at https://github.com/afastowski/frs.
BibTeX
@misc{fastowski2025from,
  author = {Alina Fastowski and Bardh Prenkaj and Gjergji Kasneci},
  title = {{From Confidence to Collapse in LLM Factual Robustness}},
  year = {2025},
  eprint = {2508.16267},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.16267},
}
Figure