Explaining Sources of Uncertainty in Automated Fact-Checking

Jingyi Sun, Greta Warren, Irina Shklovski, Isabelle Augenstein

Introduction

Large Language Models (LLMs) are increasingly used for automated fact-checking, yet their decision-making processes often remain opaque. When a model assigns a confidence score—such as 73% certainty—to a verdict, human users lack insight into the source of the remaining 27% uncertainty. This "black box" nature of uncertainty hinders trust and utility in high-stakes information verification. Existing methods typically provide raw numerical scores or vague "hedging" language that fails to explain the underlying reasoning or specific evidentiary conflicts.

The CLUE Framework

To address this gap, the authors propose CLUE (Conflict-&Agreement-aware Language-model Uncertainty Explanations). CLUE is a "plug-and-play" framework designed to provide faithful natural language explanations for model uncertainty without requiring additional fine-tuning of the underlying white-box LLM.

The pipeline consists of three primary stages:

  1. Uncertainty Scoring: The system calculates the predictive entropy of the model’s output logits, providing a mathematical measure of the model's confusion between labels (Supports, Refutes, Neutral).
  2. Conflict/Agreement Extraction: Using internal attention weights, CLUE identifies interacting spans of text between the claim and evidence. It employs the Louvain algorithm for community detection to cluster these tokens into meaningful spans, which are then labeled as agreeing, disagreeing, or unrelated.
  3. Explanation Generation: The framework uses these spans as constraints. In the CLUE-Span+Steering variant, Attention Steering (PASTA) is used to dynamically guide the model's focus toward these critical spans during the generation of the narrative explanation.
The CLUE framework identifies specific text spans that interact to create model uncertainty and translates these interactions into natural language explanations.

Experimental Evaluation

The framework was evaluated on two multi-evidence datasets: HealthVer (COVID-19 health claims) and DRUID (complex, real-world online evidence). The evaluation focused on "faithfulness"—whether the explanation accurately reflects the model's internal state—and "plausibility" to human readers.

Key Results

Quantitative analysis demonstrates that CLUE significantly improves the faithfulness of explanations compared to standard prompting baselines. As shown in the results table, CLUE-Span+Steering consistently achieves higher faithfulness correlations and Label-Explanation Entailment (LEE) scores across various models, including Qwen2.5, OLMo-2, and Gemma-2.

Quantitative performance of CLUE variants across multiple models and datasets, showing significant improvements in faithfulness and label consistency.

Human evaluations further support these findings, with participants ranking CLUE-generated explanations as more helpful, informative, and less redundant than baseline outputs.

Discussion and Limitations

A key insight from this research is the "faithfulness-plausibility trade-off." While attention steering makes explanations more faithful to the model's internal logic, it can occasionally impact the linguistic fluency of the output.

Current limitations include the framework's focus on interactions between a claim and only two evidence sources, as well as its specific targeting of uncertainty derived from conflicting evidence rather than knowledge gaps. Future work will explore scaling to larger models and more complex multi-source environments, potentially involving professional fact-checkers to validate real-world utility.

Conclusion

CLUE represents a significant step toward transparent automated fact-checking. By verbalizing the specific conflicts and agreements that drive model uncertainty, it provides users with the context necessary to interpret automated verdicts critically.

Summary updated

Abstract

Understanding sources of a model's uncertainty regarding its predictions is crucial for effective human-AI collaboration. Prior work proposes using numerical uncertainty or hedges ("I'm not sure, but ..."), which do not explain uncertainty that arises from conflicting evidence, leaving users unable to resolve disagreements or rely on the output. We introduce CLUE (Conflict-and-Agreement-aware Language-model Uncertainty Explanations), the first framework to generate natural language explanations of model uncertainty by (i) identifying relationships between spans of text that expose claim-evidence or inter-evidence conflicts and agreements that drive the model's predictive uncertainty in an unsupervised way, and (ii) generating explanations via prompting and attention steering that verbalize these critical interactions. Across three language models and two fact-checking datasets, we show that CLUE produces explanations that are more faithful to the model's uncertainty and more consistent with fact-checking decisions than prompting for uncertainty explanations without span-interaction guidance. Human evaluators judge our explanations to be more helpful, more informative, less redundant, and more logically consistent with the input than this baseline. CLUE requires no fine-tuning or architectural changes, making it plug-and-play for any white-box language model. By explicitly linking uncertainty to evidence conflicts, it offers practical support for fact-checking and generalises readily to other tasks that require reasoning over complex information.
BibTeX
@inproceedings{sun2026explaining,
  author = {Jingyi Sun and Greta Warren and Irina Shklovski and Isabelle Augenstein},
  title = {{Explaining Sources of Uncertainty in Automated Fact-Checking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2110},
}
Figure