Strong and Light Baseline Models for Fact-Checking Joint Inference

Kateryna Tymoshenko, Alessandro MoschittiACL 2021

TL;DR

This paper introduces surprisingly effective and computationally light baseline models for joint fact-checking inference. By simplifying complex graph-based approaches and using basic pooling or weighted sum operations on Transformer embeddings, they achieve state-of-the-art results on the FEVER benchmark, demonstrating that simpler methods can be highly competitive.

What Problem Does This Paper Address?

Fact-checking, especially in the context of verifying claims against large amounts of textual evidence, is a crucial but challenging task. Existing state-of-the-art (SOTA) methods often rely on complex graph neural networks to model interactions between multiple pieces of evidence. While these methods can be powerful, they come with significant computational and memory overhead. This paper aims to address the gap between the complexity of current SOTA models and the need for simpler, more efficient, yet still highly accurate methods for joint inference over evidence. They specifically focus on the Evidence Reasoning (ER) component of fact-checking, where the goal is to predict whether a claim is supported, refuted, or if there's not enough information, given a set of retrieved evidence sentences.

Methods

The authors propose four lightweight baseline models for joint inference:

  • Local: This model treats each evidence sentence independently, predicts a label and confidence score for each, and then aggregates these predictions. It uses heuristics to combine evidence labels.
  • Concat: This approach concatenates the claim and each evidence sentence into a single input sequence for a Transformer model (like BERT or RoBERTa).
  • MaxPool: This method encodes each claim-evidence pair individually, then applies max-pooling across the resulting embeddings to get a single representation for the entire set of evidence.
  • WgtSum: Similar to MaxPool, this model encodes each claim-evidence pair but then computes a weighted sum of these embeddings. The weights are intended to reflect the importance of each evidence piece.

The core innovation lies in demonstrating that these simple aggregation strategies (max pooling, weighted sum, concatenation) applied to Transformer-based embeddings can achieve SOTA performance, challenging the necessity of complex graph structures.

Datasets

The primary dataset used for training and evaluation is the FEVER 1.0 Shared Task dataset. This dataset consists of claims and associated evidence sentences from Wikipedia, labeled as Supported (SUP), Refuted (REF), or Not Enough Information (NEI). The authors also mention using evidence sentences retrieved by Liu et al. (2020) for their ER experiments, which were obtained using a specific document retrieval (DocIR) and evidence selection (ES) pipeline.

Key Insights and Ideas

  • Simplicity can be SOTA: The most striking insight is that simple aggregation methods (max pooling, weighted sum) applied to Transformer embeddings can match or exceed the performance of much more complex graph-based models on the FEVER benchmark.
  • Transformer Embeddings are Powerful: The paper highlights the rich semantic information captured by Transformer models, which can be effectively leveraged with basic pooling or summation techniques for complex reasoning tasks.
  • Joint Inference is Key, but Complexity Isn't Always Necessary: While joint reasoning over multiple evidence pieces is beneficial, the authors show that sophisticated graph structures are not always required to achieve this.
  • Baselines Matter: The paper emphasizes the importance of strong, reproducible baselines to guide research and prevent the community from pursuing overly complex solutions when simpler ones suffice.

Key Results

  • The proposed MaxPool and WgtSum models, when using RoBERTa-large, achieved state-of-the-art results on the FEVER DEV set, outperforming previously published SOTA systems.
  • On the official FEVER leaderboard as of June 1st, 2021, their WgtSum model was ranked seventh in FEVER score and sixth in Label Accuracy (LA).
  • The absolute difference from the fourth top LA was only 0.2%, indicating their models are highly competitive.
  • The authors' simple baselines achieved comparable performance to KGAT (a complex graph-based model) with significantly fewer trainable parameters in the ER component (around 3K vs. over 1.3 million for KGAT with RoBERTa-large).

Limitations and Future Work

  • Overfitting on DEV: The authors acknowledge that their strong performance on the DEV set might indicate some overfitting, and they are not able to compare directly with the top submissions on the blind TEST set due to unknown methodologies.
  • DocIR/ES Dependency: The performance of their ER models is dependent on the quality of the evidence retrieved by the DocIR and ES components. While they used a strong ES system, they note that their FEVER score might be lower than some SOTA systems if their evidence recall is not as high.
  • Future Work: The authors plan to leverage their findings to build systems that can effectively trade off efficiency for accuracy, suggesting further exploration into optimizing the balance between model complexity and performance. They also note that their models are easily reproducible and can serve as strong baselines for future research.

Summary updated

BibTeX
@inproceedings{tymoshenko2021strong,
  author = {Kateryna Tymoshenko and Alessandro Moschitti},
  title = {{Strong and Light Baseline Models for Fact-Checking Joint Inference}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure