SAFE: Structured Argumentation for Fact-checking with Explanations

Xiaoou Wang, Elena Cabrio, Serena VillataIJCAI 2025 · DOI

TL;DR

This paper introduces SAFE, a novel framework for automated fact-checking that goes beyond simple claim verification by generating structured, argument-based explanations. SAFE can produce these explanations from existing fact-checking articles or, more importantly, by retrieving and summarizing evidence from a corpus when no human-written article is available, making it a more practical solution for real-world disinformation challenges.

What Problem Does This Paper Address?

The proliferation of online misinformation necessitates robust fact-checking systems. While automated fact-checking has advanced, a critical component often missing is the explanation – the justification for why a claim is deemed true or false. Existing methods often rely on human-written fact-checking articles, which are not always available for every new claim. This creates a gap in providing timely and comprehensive justifications, leaving users without the necessary context to understand and trust the fact-checking outcome. Furthermore, "black-box" models without explanations can sometimes lead to a "backfire effect," reinforcing incorrect beliefs.

Methods

SAFE tackles this problem with a three-pronged approach:

  1. Argument-Structured Summarization: This module generates summaries that mirror how human fact-checkers analyze claims and evidence. It breaks down the information into premises that support, attack, or partially support/attack the main claim. This is achieved by fine-tuning a language model (Mixtral-8x22B) on a specialized dataset (LIARArg).
  2. Retrieval-based Summarization: This is a key innovation. When a human-written fact-checking article is unavailable, SAFE can retrieve relevant evidence from a corpus (e.g., ExClaim) and then generate an argument-structured summary based on this retrieved evidence. This module jointly optimizes evidence retrieval and summarization, using Atlas as a backbone for retrieval-augmented generation.
  3. Claim Verification: This module assesses the truthfulness of a claim by analyzing it in conjunction with the generated structured summary. It leverages a state-of-the-art fake news classification method, incorporating entity graphs and graph attention networks for enhanced analysis.

Datasets

  • LIARArg: Used for fine-tuning the argument-structured summarization module. This dataset is a subset of LIAR-PLUS.
  • LIAR-PLUS, FNC-1, Check-COVID: These are benchmark datasets used for evaluating the claim verification module.
  • ExClaim: Used for evaluating the retrieval-based summarization module.

Key Insights and Ideas

  • Argument structure is crucial for fact-checking: Simply summarizing evidence isn't enough; understanding the relationship between evidence and a claim (support, attack) is vital for effective automated fact-checking.
  • Bridging the gap for unseen claims: The retrieval-based summarization is a significant step towards practical, real-world fact-checking, as it doesn't require pre-existing human-written articles for every claim.
  • Structured explanations outperform raw ROUGE scores: While ROUGE scores measure textual overlap, the paper demonstrates that argument-structured summaries lead to better claim verification performance, even if ROUGE scores are not always higher. This highlights the importance of semantic understanding over surface-level similarity.

Key Results

  • Improved Claim Verification: SAFE's argument-structured summaries significantly improve the performance of claim verification modules compared to other summarization methods, even outperforming human-written summaries on some benchmarks (e.g., LIAR-PLUS and FNC-1).
  • Competitive Retrieval-Summarization: On the ExClaim benchmark, SAFE achieved a higher F1 score (0.74) for claim verification compared to JustiLM (0.67), a state-of-the-art retrieval-summarization method. Notably, SAFE's performance was comparable to using human-written summaries (0.72 F1 score).
  • Top-k Recall: SAFE demonstrated strong top-k recall performance on the ExClaim dataset, outperforming JustiLM across various k values (5, 10, 15, 20).

Limitations and Future Work

  • Limitations: The paper acknowledges that while SAFE's structured summaries can be more effective than human-written ones for claim verification, the ROUGE scores for these summaries might not always be higher. This reinforces the idea that overlap-based metrics are insufficient for evaluating fact-checking-oriented summarization.
  • Future Work: The authors plan to:
  • Incorporate argument schemes for more in-depth explanations.
  • Enhance the retrieval-summarization module by leveraging external knowledge bases.
  • Investigate the factual consistency of the generated explanations.
  • Diversify training corpora to improve generalization across different domains.

Summary updated

BibTeX
@inproceedings{wang2025safe,
  author = {Xiaoou Wang and Elena Cabrio and Serena Villata},
  title = {{SAFE: Structured Argumentation for Fact-checking with Explanations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/1274},
}
Figure