Abstract
The LisT5 framework redefines fact verification by transitioning from traditional "pointwise" evidence checking to a "listwise" approach that reasons over multiple pieces of evidence simultaneously. By leveraging the T5 transformer architecture and a novel data augmentation strategy that introduces noise during training, LisT5 achieved a state-of-the-art FEVER score of 75.87%, topping the leaderboard at the time of publication.
Introduction
Automated fact verification is typically structured as a three-stage pipeline: document retrieval, sentence selection, and veracity label prediction (Supported, Refuted, or Not Enough Info). A significant bottleneck often occurs in the final stage due to the prevalence of "pointwise" approaches. These systems evaluate claims against individual sentences and aggregate scores, often failing to capture the "joint reasoning" required for complex claims where truth emerges only from the synthesis of multiple evidence pieces. Furthermore, models trained exclusively on clean "gold" evidence frequently underperform when faced with the noisy outputs of real-world retrieval systems.
Methods
The researchers introduced LisT5, built on the Text-to-Text Transfer Transformer (T5) architecture. While following the standard three-stage pipeline, LisT5 incorporates key innovations in sentence selection and label prediction.
Sentence Selection
The system utilizes monoT5 as a reranker. A specific enhancement involves appending Named Entities extracted from the claim into the prompt, which directs the model's focus toward documents whose titles align with the claim's entities.
Label Prediction: The Listwise Approach
The core innovation of LisT5 lies in its label prediction strategy:
- Listwise Concatenation: Instead of individual scoring, the top five candidate sentences are concatenated into a single input string. This enables the transformer’s self-attention mechanism to process all evidence holistically.
- Listwise Data Augmentation: To ensure robustness, the training set is "infused" with noise—randomly selected non-evidence sentences from retrieved documents. This strategy bridges the gap between clean training data and the imperfect conditions of inference.
Datasets
The framework was evaluated using the FEVER (Fact Extraction and VERification) dataset, which contains 185,000 claims derived from Wikipedia. Claims are categorized as SUPPORTS, REFUTES, or NOT ENOUGH INFO. Performance is measured by the FEVER score, requiring both a correct label and the identification of accurate supporting evidence.
Key Results
LisT5 demonstrated dominant performance on the FEVER task, achieving a FEVER score of 75.87% on the blind test set.
Insights from Ablation and Analysis
- Contextual Reasoning: The listwise approach consistently outperformed pointwise aggregation methods (such as sum or max pooling), indicating T5's effectiveness in capturing inter-sentence semantics.
- The Impact of Augmentation: Data augmentation proved to be the "secret sauce." Training with a mix of gold evidence and distractor sentences improved label accuracy (LA) by over 10 points compared to training on gold evidence alone (81.26% vs 70.66% on the Dev set).
- Joint Reasoning Capability: On claims requiring at least two pieces of evidence, the model achieved 78.02% accuracy, validating its ability to synthesize information across multiple sentences.
Limitations and Future Work
Error analysis of 200 incorrect predictions identified two primary challenges:
- Semantic Ambiguity: The model occasionally fails to distinguish between logically distinct but semantically similar terms, such as "retired" versus "resigned."
- Data Quality: Some errors stemmed from "noisy" labels or missing evidence within the FEVER dataset itself rather than model failure.
Future research may explore more sophisticated handling of semantic nuances and the application of listwise reasoning to other complex NLP tasks, such as multi-hop question answering.
Summary updated