Team Alex at CLEF CheckThat! 2020: Identifying Check-Worthy Tweets With Transformer Models

Alex Nikolov, Giovanni Da San Martino, Ivan Koychev, Preslav Nakov

Abstract

While misinformation and disinformation have been thriving in social media for years, with the emergence of the COVID-19 pandemic, the political and the health misinformation merged, thus elevating the problem to a whole new level and giving rise to the first global infodemic. The fight against this infodemic has many aspects, with fact-checking and debunking false and misleading claims being among the most important ones. Unfortunately, manual fact-checking is time-consuming and automatic fact-checking is resource-intense, which means that we need to pre-filter the input social media posts and to throw out those that do not appear to be check-worthy. With this in mind, here we propose a model for detecting check-worthy tweets about COVID-19, which combines deep contextualized text representations with modeling the social context of the tweet. Our official submission to the English version of CLEF-2020 CheckThat! Task 1, system Team_Alex, was ranked second with a MAP score of 0.8034.

Introduction

In the early months of 2020, the world faced what the WHO termed an "infodemic"—a massive surge of information, much of it false or misleading, spreading rapidly across social media. Fact-checking organizations were overwhelmed, creating an urgent need for automated "triage" systems. The goal of check-worthiness estimation is to identify which social media posts deserve the immediate attention of human fact-checkers based on their factual nature, potential for harm, and likelihood of being false.

Methodology

The "Corona Hack"

A significant challenge in 2020 was that state-of-the-art NLP models like BERT were trained on data from 2019 and earlier. Consequently, these models lacked "COVID-19" in their vocabulary. To address this temporal shift, Team Alex implemented a preprocessing strategy where terms like covid-19 and coronavirus were replaced with ebola. Since ebola was already in the models' pre-trained vocabularies and carries similar semantic weight as a viral outbreak, this allowed the models to leverage prior knowledge to understand the context of the tweets.

Model Selection and Baselines

The team experimented with a wide range of text representations, from traditional word embeddings to modern Transformer architectures.

Performance comparison of traditional embeddings and Transformer models across different preprocessing strategies.

As shown in the preliminary experiments, Transformer-based models consistently outperformed traditional pooling methods using GloVe or FastText. Among the Transformers, RoBERTa-base emerged as the most effective balance of performance and computational efficiency.

Social Context and Metadata

Beyond text, the system incorporated 12 additional features to capture the social context:

  • Engagement: Retweet and follower counts.
  • Account Authority: Verification status and account age.
  • Source Factuality: Boolean features indicating if a tweet linked to a "high factuality" vs. "conspiracy/fake news" outlet, based on Media Bias/Fact Check judgments.

These features were combined with Transformer predictions using a Logistic Regression ensemble layer.

Results and Analysis

Detailed comparison of Transformer architectures under various preprocessing conditions.

The experimental results highlight the impact of different preprocessing strategies. The "Corona" strategy (the Ebola substitution) and the "SW+C" strategy (a custom stop-word list that preserves personal and demonstrative pronouns) both provided performance gains over default settings. The preservation of pronouns like "he" or "this" proved vital, as they often point to the subject of a verifiable claim.

Key Performance Metrics

  • Official MAP: 0.8034 (2nd place globally).
  • Precision at the Top: The model achieved a P@1, P@3, and P@5 of 1.0000, indicating perfect accuracy for the highest-ranked tweets.
  • Metadata Impact: The inclusion of source factuality features significantly improved macro-F1 scores, helping the model distinguish borderline cases.

Conclusion

Team Alex's success demonstrates that while deep contextualized representations are powerful, pragmatic engineering—such as mapping new concepts to known ones and incorporating social metadata—is essential for real-world NLP tasks during rapidly evolving events. Future work will explore more sophisticated social context modeling, including network sharing structures and newer architectures like T5 or Electra.

Summary updated

Abstract

While misinformation and disinformation have been thriving in social media for years, with the emergence of the COVID-19 pandemic, the political and the health misinformation merged, thus elevating the problem to a whole new level and giving rise to the first global infodemic. The fight against this infodemic has many aspects, with fact-checking and debunking false and misleading claims being among the most important ones. Unfortunately, manual fact-checking is time-consuming and automatic fact-checking is resource-intense, which means that we need to pre-filter the input social media posts and to throw out those that do not appear to be check-worthy. With this in mind, here we propose a model for detecting check-worthy tweets about COVID-19, which combines deep contextualized text representations with modeling the social context of the tweet. We further describe a number of additional experiments and comparisons, which we believe should be useful for future research as they provide some indication about what techniques are effective for the task. Our official submission to the English version of CLEF-2020 CheckThat! Task 1, system Team_Alex, was ranked second with a MAP score of 0.8034, which is almost tied with the wining system, lagging behind by just 0.003 MAP points absolute.
BibTeX
@inproceedings{nikolov2020team,
  author = {Alex Nikolov and Giovanni Da San Martino and Ivan Koychev and Preslav Nakov},
  title = {{Team Alex at CLEF CheckThat! 2020: Identifying Check-Worthy Tweets With Transformer Models}},
  booktitle = {CLEF (Working Notes)},
  year = {2020},
  url = {https://openreview.net/forum?id=oO825Hxo-br},
}
Figure