Distilling the Evidence to Augment Fact Verification Models

Beatrice Portelli, Jason Zhao, Tal Schuster, Giuseppe Serra, Enrico Santus2020 · DOI

Abstract

The alarming spread of fake news in social media, together with the impossibility of scaling manual fact verification, motivated the development of natural language processing techniques to automatically verify the veracity of claims. Most approaches perform a claim-evidence classification without providing any insights about why the claim is trustworthy or not. We propose, instead, a model-agnostic framework that consists of two modules: (1) a span extractor, which identifies the crucial information connecting claim and evidence; and (2) a classifier that combines claim, evidence, and the extracted spans to predict the veracity of the claim. We show that the spans are informative for the classifier, improving performance and robustness. Tested on several state-of-the-art models over the FEVER dataset, the enhanced classifiers consistently achieve higher accuracy while also showing reduced sensitivity to artifacts in the claims.

Introduction

Automated fact verification—determining if a claim is supported or refuted by a piece of evidence—is a critical tool in the fight against misinformation. However, current natural language processing (NLP) models face two major hurdles. First is a lack of transparency: most models are "black boxes" that output a verdict without explaining which part of the evidence led to that conclusion. Second is the presence of dataset artifacts: models often "cheat" by picking up on linguistic patterns or "give-away phrases" in the claims themselves rather than actually reasoning about the relationship between the claim and the evidence. This research proposes a framework to solve these issues by forcing models to identify and utilize specific "polarizing" spans of text.

Methodology

The authors introduce a two-module pipeline designed to be model-agnostic, allowing it to enhance various existing classification architectures.

  1. Span Extractor ($M_{span}$): This module utilizes a TokenMasker architecture (based on RoBERTa) to identify the minimal group of tokens in the evidence necessary to determine a verdict. It is pretrained on the Multi-Genre Natural Language Inference (MNLI) task to recognize logical relationships.
  2. Classification Module ($M_{classifier}$): The extracted spans are concatenated with the original evidence and fed into a classifier. By highlighting the most important parts of the evidence, the system provides a "rationale" that the classifier uses to make a more informed decision.
The proposed model-agnostic framework featuring a Span Extractor and a Classification Module.

Experimental Setup

The framework was evaluated using three high-performing architectures—BiDAF (Bi-Directional Attention Flow), NSMN (Neural Semantic Matching Networks), and BERT—across two primary datasets:

  • FEVER: The standard large-scale dataset for Fact Extraction and VERification, containing over 185,000 Wikipedia-derived claims.
  • SYMMETRIC FEVER: A more challenging subset designed to remove "give-away phrases" in claims, forcing models to rely on actual evidence reasoning rather than claim-side artifacts.

Results and Analysis

The inclusion of distilled spans consistently improved performance across all tested models and datasets. As shown in the results table, the "informed" models (using spans) outperformed the baseline "uninformed" models.

Performance comparison of BiDAF, NSMN, and BERT models with and without distilled spans on FEVER and SYMMETRIC test sets.

Key findings include:

  • Accuracy Boost: On the standard FEVER dataset, BiDAF accuracy rose from 73.90% to 75.12%.
  • Robustness on Symmetric Data: Gains were even more pronounced on the SYMMETRIC FEVER dataset, where BiDAF saw a jump from 49.16% to 52.24%. This suggests the spans help the model ignore misleading artifacts.
  • Human-Centric Validation: In a manual evaluation of 100 claim-evidence pairs, humans found the extracted spans to be "well explanatory" in 75% of cases (88% for refuted claims and 67% for supported claims).

Conclusion

This research demonstrates that distilling evidence into key spans provides a dual benefit: it offers human-readable justifications for model verdicts and improves overall classification accuracy and robustness. While the gains on massive pre-trained models like BERT are more modest, the framework provides a valuable tool for building more transparent and reliable fact-checking systems.

Summary updated

Abstract

The alarming spread of fake news in social media, together with the impossibility of scaling manual fact verification, motivated the development of natural language processing techniques to automatically verify the veracity of claims. Most approaches perform a claimevidence classification without providing any insights about why the claim is trustworthy or not. We propose, instead, a model-agnostic framework that consists of two modules: (1) a span extractor, which identifies the crucial information connecting claim and evidence; and (2) a classifier that combines claim, evidence, and the extracted spans to predict the veracity of the claim. We show that the spans are informative for the classifier, improving performance and robustness. Tested on several state-of-the-art models over the FEVER dataset, the enhanced classifiers consistently achieve higher accuracy while also showing reduced sensitivity to artifacts in the claims.
BibTeX
@misc{portelli2020distilling,
  author = {Beatrice Portelli and Jason Zhao and Tal Schuster and Giuseppe Serra and Enrico Santus},
  title = {{Distilling the Evidence to Augment Fact Verification Models}},
  year = {2020},
  doi = {10.18653/v1/2020.fever-1.7},
}
Figure