Abstract
While misinformation and disinformation have been thriving in social media for years, with the emergence of the COVID-19 pandemic, the political and the health misinformation merged, thus elevating the problem to a whole new level and giving rise to the first global infodemic. The fight against this infodemic has many aspects, with fact-checking and debunking false and misleading claims being among the most important ones. Unfortunately, manual fact-checking is time-consuming and automatic fact-checking is resource-intense, which means that we need to pre-filter the input social media posts and to throw out those that do not appear to be check-worthy. With this in mind, here we propose a model for detecting check-worthy tweets about COVID-19, which combines deep contextualized text representations with modeling the social context of the tweet. Our official submission to the English version of CLEF-2020 CheckThat! Task 1, system Team_Alex, was ranked second with a MAP score of 0.8034.
Introduction
In the early months of 2020, the world faced what the WHO termed an "infodemic"—a massive surge of information, much of it false or misleading, spreading rapidly across social media. Fact-checking organizations were overwhelmed, creating an urgent need for automated "triage" systems. The goal of check-worthiness estimation is to identify which social media posts deserve the immediate attention of human fact-checkers based on their factual nature, potential for harm, and likelihood of being false.
Methodology
The "Corona Hack"
A significant challenge in 2020 was that state-of-the-art NLP models like BERT were trained on data from 2019 and earlier. Consequently, these models lacked "COVID-19" in their vocabulary. To address this temporal shift, Team Alex implemented a preprocessing strategy where terms like covid-19 and coronavirus were replaced with ebola. Since ebola was already in the models' pre-trained vocabularies and carries similar semantic weight as a viral outbreak, this allowed the models to leverage prior knowledge to understand the context of the tweets.
Model Selection and Baselines
The team experimented with a wide range of text representations, from traditional word embeddings to modern Transformer architectures.
As shown in the preliminary experiments, Transformer-based models consistently outperformed traditional pooling methods using GloVe or FastText. Among the Transformers, RoBERTa-base emerged as the most effective balance of performance and computational efficiency.
Social Context and Metadata
Beyond text, the system incorporated 12 additional features to capture the social context:
- Engagement: Retweet and follower counts.
- Account Authority: Verification status and account age.
- Source Factuality: Boolean features indicating if a tweet linked to a "high factuality" vs. "conspiracy/fake news" outlet, based on Media Bias/Fact Check judgments.
These features were combined with Transformer predictions using a Logistic Regression ensemble layer.
Results and Analysis
The experimental results highlight the impact of different preprocessing strategies. The "Corona" strategy (the Ebola substitution) and the "SW+C" strategy (a custom stop-word list that preserves personal and demonstrative pronouns) both provided performance gains over default settings. The preservation of pronouns like "he" or "this" proved vital, as they often point to the subject of a verifiable claim.
Key Performance Metrics
- Official MAP: 0.8034 (2nd place globally).
- Precision at the Top: The model achieved a P@1, P@3, and P@5 of 1.0000, indicating perfect accuracy for the highest-ranked tweets.
- Metadata Impact: The inclusion of source factuality features significantly improved macro-F1 scores, helping the model distinguish borderline cases.
Conclusion
Team Alex's success demonstrates that while deep contextualized representations are powerful, pragmatic engineering—such as mapping new concepts to known ones and incorporating social metadata—is essential for real-world NLP tasks during rapidly evolving events. Future work will explore more sophisticated social context modeling, including network sharing structures and newer architectures like T5 or Electra.
Summary updated