Introduction
Large Language Models (LLMs) are increasingly used for automated fact-checking, yet their decision-making processes often remain opaque. When a model assigns a confidence score—such as 73% certainty—to a verdict, human users lack insight into the source of the remaining 27% uncertainty. This "black box" nature of uncertainty hinders trust and utility in high-stakes information verification. Existing methods typically provide raw numerical scores or vague "hedging" language that fails to explain the underlying reasoning or specific evidentiary conflicts.
The CLUE Framework
To address this gap, the authors propose CLUE (Conflict-&Agreement-aware Language-model Uncertainty Explanations). CLUE is a "plug-and-play" framework designed to provide faithful natural language explanations for model uncertainty without requiring additional fine-tuning of the underlying white-box LLM.
The pipeline consists of three primary stages:
- Uncertainty Scoring: The system calculates the predictive entropy of the model’s output logits, providing a mathematical measure of the model's confusion between labels (Supports, Refutes, Neutral).
- Conflict/Agreement Extraction: Using internal attention weights, CLUE identifies interacting spans of text between the claim and evidence. It employs the Louvain algorithm for community detection to cluster these tokens into meaningful spans, which are then labeled as agreeing, disagreeing, or unrelated.
- Explanation Generation: The framework uses these spans as constraints. In the CLUE-Span+Steering variant, Attention Steering (PASTA) is used to dynamically guide the model's focus toward these critical spans during the generation of the narrative explanation.
Experimental Evaluation
The framework was evaluated on two multi-evidence datasets: HealthVer (COVID-19 health claims) and DRUID (complex, real-world online evidence). The evaluation focused on "faithfulness"—whether the explanation accurately reflects the model's internal state—and "plausibility" to human readers.
Key Results
Quantitative analysis demonstrates that CLUE significantly improves the faithfulness of explanations compared to standard prompting baselines. As shown in the results table, CLUE-Span+Steering consistently achieves higher faithfulness correlations and Label-Explanation Entailment (LEE) scores across various models, including Qwen2.5, OLMo-2, and Gemma-2.
Human evaluations further support these findings, with participants ranking CLUE-generated explanations as more helpful, informative, and less redundant than baseline outputs.
Discussion and Limitations
A key insight from this research is the "faithfulness-plausibility trade-off." While attention steering makes explanations more faithful to the model's internal logic, it can occasionally impact the linguistic fluency of the output.
Current limitations include the framework's focus on interactions between a claim and only two evidence sources, as well as its specific targeting of uncertainty derived from conflicting evidence rather than knowledge gaps. Future work will explore scaling to larger models and more complex multi-source environments, potentially involving professional fact-checkers to validate real-world utility.
Conclusion
CLUE represents a significant step toward transparent automated fact-checking. By verbalizing the specific conflicts and agreements that drive model uncertainty, it provides users with the context necessary to interpret automated verdicts critically.
Summary updated