Overview of the CLEF-2022 CheckThat! Lab Task 2 on Detecting Previously Fact-Checked Claims

Preslav Nakov, Giovanni Da San Martino, Firoj Alam, Shaden Shaar, Hamdy Mubarak, Nikolay BabulkovCLEF (Working Notes) 2022 · OpenReview

TL;DR

This paper presents Task 2 of the CLEF 2022 CheckThat! lab, which focuses on automatically identifying previously fact-checked claims within social media posts and political speeches. The goal is to reduce the manual effort in fact-checking by surfacing existing verifications, with six teams participating and utilizing transformer-based models to achieve improvements over baselines.

What Problem Does This Paper Address?

The proliferation of misinformation and disinformation online necessitates efficient and scalable fact-checking systems. However, these systems often face credibility issues, and manual fact-checking is time-consuming. This paper addresses the challenge of automating a crucial step in the fact-checking pipeline: detecting whether a given claim has already been fact-checked. By identifying previously verified claims, human fact-checkers can be more efficient, and automated systems can be more robust.

Methods

The core of this work is the CLEF 2022 CheckThat! Lab Task 2, which is framed as a ranking problem. Given an input claim (from a tweet or speech), systems are tasked with ranking a collection of previously fact-checked claims based on their relevance and usefulness for fact-checking the input.

Two subtasks were defined:

  • Subtask 2A: Detecting previously fact-checked claims in tweets. This subtask was offered in Arabic and English.
  • Subtask 2B: Detecting previously fact-checked claims in political debates or speeches. This subtask was offered in English.

Participating systems primarily employed transformer-based models, such as BERT, RoBERTa, Sentence-BERT, ST5, and GPT-Neo. Approaches included:

  • Semantic and lexical similarity measures: Calculating similarity between input claims and verified claims using embeddings and lexical features.
  • Re-ranking modules: Using models to refine initial rankings based on semantic and lexical information.
  • Ensemble approaches: Combining multiple models or techniques.
  • Data augmentation: Techniques like back-translation were used to improve model performance.
  • Preprocessing: Steps like removing URLs, @-symbols, and user information were common.

Datasets

The paper describes the datasets created for the CLEF 2022 CheckThat! lab, specifically for Task 2.

  • Arabic Dataset: Consists of 908 tweets, matched against 1,089 verified claims. This data was derived from AraFacts [29] and ClaimsKG [30], with translations to Arabic. The collection of previously fact-checked claims for Arabic contains 30,379 entries.
  • English Dataset: Utilizes claims from political debates [1] and tweets. For tweets, 1,610 annotated tweets were collected, each linked to a claim from the Snopes fact-checking website. The collection of verified claims from Snopes comprises 13,835 entries. For political debates, 752 claims were matched against 869 verified claims from PolitiFact.

The paper also references datasets from previous editions of the CheckThat! lab [6, 7].

Key Insights and Ideas

  • Transformer dominance: Transformer-based models, particularly those fine-tuned on task-specific data or pre-trained on large corpora, were the backbone of most successful approaches.
  • Re-ranking is crucial: While initial retrieval of relevant claims is important, re-ranking them using more sophisticated models significantly improves performance.
  • Multilinguality is a growing focus: The inclusion of Arabic alongside English highlights the increasing importance of developing fact-checking tools for diverse linguistic contexts.
  • Combining semantic and lexical features: Even with powerful transformer models, combining semantic similarity (from embeddings) with lexical similarity (e.g., token overlap) can yield better results.
  • The value of existing work: The task builds upon previous efforts in fact-checking and claim detection, leveraging existing datasets and annotation methodologies.

Key Results

  • Subtask 2A (English): Six teams participated, and all improved upon the random baseline. The top-performing system, RIET Lab, utilized a sentence transformer (sentence-t5) for candidate selection and a generative model (GPT-Neo) for re-ranking. Team AI Rational secured second place.
  • Subtask 2A (Arabic): Only one team, BigIR, participated. Their system achieved a performance of 0.0, similar to the random baseline, indicating challenges in this language setting or with the specific dataset.
  • Subtask 2B (English): One team, SimBa, participated and achieved a MAP@5 score of 0.459, outperforming the random baseline. Their approach involved computing various similarity measures between input and verified claims.

Limitations and Future Work

  • Arabic performance: The low performance in the Arabic subtask suggests that current transformer models and training data may not be as effective for Arabic fact-checking compared to English.
  • Data scarcity: While new datasets were created, the overall volume of annotated data for previously fact-checked claims might still be a limiting factor for achieving very high performance.
  • Future directions: The authors suggest exploring other tasks relevant to journalistic and social media analysis, such as coverage bias in news and subjectivity detection, to further enhance media literacy and critical thinking. They also aim to continue promoting multilinguality in future iterations of the lab.

Summary updated

BibTeX
@inproceedings{nakov2022overview,
  author = {Preslav Nakov and Giovanni Da San Martino and Firoj Alam and Shaden Shaar and Hamdy Mubarak and Nikolay Babulkov},
  title = {{Overview of the CLEF-2022 CheckThat! Lab Task 2 on Detecting Previously Fact-Checked Claims}},
  booktitle = {CLEF (Working Notes)},
  year = {2022},
  url = {https://openreview.net/forum?id=jfdHaEzVyo},
}
Figure