Exploring Listwise Evidence Reasoning with T5 for Fact Verification

Kelvin Jiang, Ronak Pradeep, Jimmy Lin

Abstract

The LisT5 framework redefines fact verification by transitioning from traditional "pointwise" evidence checking to a "listwise" approach that reasons over multiple pieces of evidence simultaneously. By leveraging the T5 transformer architecture and a novel data augmentation strategy that introduces noise during training, LisT5 achieved a state-of-the-art FEVER score of 75.87%, topping the leaderboard at the time of publication.

Introduction

Automated fact verification is typically structured as a three-stage pipeline: document retrieval, sentence selection, and veracity label prediction (Supported, Refuted, or Not Enough Info). A significant bottleneck often occurs in the final stage due to the prevalence of "pointwise" approaches. These systems evaluate claims against individual sentences and aggregate scores, often failing to capture the "joint reasoning" required for complex claims where truth emerges only from the synthesis of multiple evidence pieces. Furthermore, models trained exclusively on clean "gold" evidence frequently underperform when faced with the noisy outputs of real-world retrieval systems.

Methods

The researchers introduced LisT5, built on the Text-to-Text Transfer Transformer (T5) architecture. While following the standard three-stage pipeline, LisT5 incorporates key innovations in sentence selection and label prediction.

Sentence Selection

The system utilizes monoT5 as a reranker. A specific enhancement involves appending Named Entities extracted from the claim into the prompt, which directs the model's focus toward documents whose titles align with the claim's entities.

Performance of sentence selection methods, highlighting the impact of Named Entity Recognition (NER) in the monoT5 model.

Label Prediction: The Listwise Approach

The core innovation of LisT5 lies in its label prediction strategy:

  1. Listwise Concatenation: Instead of individual scoring, the top five candidate sentences are concatenated into a single input string. This enables the transformer’s self-attention mechanism to process all evidence holistically.
  2. Listwise Data Augmentation: To ensure robustness, the training set is "infused" with noise—randomly selected non-evidence sentences from retrieved documents. This strategy bridges the gap between clean training data and the imperfect conditions of inference.

Datasets

The framework was evaluated using the FEVER (Fact Extraction and VERification) dataset, which contains 185,000 claims derived from Wikipedia. Claims are categorized as SUPPORTS, REFUTES, or NOT ENOUGH INFO. Performance is measured by the FEVER score, requiring both a correct label and the identification of accurate supporting evidence.

Key Results

LisT5 demonstrated dominant performance on the FEVER task, achieving a FEVER score of 75.87% on the blind test set.

Comparison of LisT5 variants against state-of-the-art methods on the FEVER dataset.

Insights from Ablation and Analysis

  • Contextual Reasoning: The listwise approach consistently outperformed pointwise aggregation methods (such as sum or max pooling), indicating T5's effectiveness in capturing inter-sentence semantics.
  • The Impact of Augmentation: Data augmentation proved to be the "secret sauce." Training with a mix of gold evidence and distractor sentences improved label accuracy (LA) by over 10 points compared to training on gold evidence alone (81.26% vs 70.66% on the Dev set).
  • Joint Reasoning Capability: On claims requiring at least two pieces of evidence, the model achieved 78.02% accuracy, validating its ability to synthesize information across multiple sentences.

Limitations and Future Work

Error analysis of 200 incorrect predictions identified two primary challenges:

  1. Semantic Ambiguity: The model occasionally fails to distinguish between logically distinct but semantically similar terms, such as "retired" versus "resigned."
  2. Data Quality: Some errors stemmed from "noisy" labels or missing evidence within the FEVER dataset itself rather than model failure.

Future research may explore more sophisticated handling of semantic nuances and the application of listwise reasoning to other complex NLP tasks, such as multi-hop question answering.

Summary updated

Abstract

This work explores a framework for fact verification that leverages pretrained sequence-to-sequence transformer models for sentence selection and label prediction, two key sub-tasks in fact verification. Most notably, improving on previous pointwise aggregation approaches for label prediction, we take advantage of T5 using a listwise approach coupled with data augmentation. With this enhancement, we observe that our label prediction stage is more robust to noise and capable of verifying complex claims by jointly reasoning over multiple pieces of evidence. Experimental results on the FEVER task show that our system attains a FEVER score of 75.87% on the blind test set. This puts our approach atop the competitive FEVER leaderboard at the time of our work, scoring higher than the second place submission by almost two points in label accuracy and over one point in FEVER score.
BibTeX
@inproceedings{jiang2021exploring,
  author = {Kelvin Jiang and Ronak Pradeep and Jimmy Lin},
  title = {{Exploring Listwise Evidence Reasoning with T5 for Fact Verification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.51},
}
Figure