TL;DR
Researchers from QMUL and partner institutions developed a system to automatically rank COVID-19 tweets by their "check-worthiness"—essentially identifying which claims are most important for fact-checkers to investigate. By combining a domain-specific language model (CT-BERT) with a convolutional neural network (CNN) and specialized tokenization for numeric expressions, their system achieved 4th place in the CLEF 2020 CheckThat! competition.
What Problem Does This Paper Address?
During the COVID-19 pandemic, social media became a primary source of information, but also a breeding ground for harmful misinformation and conspiracy theories. Fact-checking organizations are often overwhelmed by the sheer volume of posts.
The challenge isn't just identifying if a tweet is true or false, but determining its check-worthiness: Does it contain a claim that is of interest to a large audience and potentially harmful if left unverified? This paper tackles the task of automatically prioritizing these high-stakes tweets so that human fact-checkers can focus their limited resources where they matter most.
Methods
The authors proposed a deep learning pipeline centered around a 1D Convolutional Neural Network (CNN) that processes word-level vector representations.
Key components of their approach include:
- Domain-Specific Embeddings: They compared standard BERT and ELMo against COVID-Twitter-BERT (CT-BERT), a model pre-trained specifically on COVID-19 related tweets. CT-BERT significantly outperformed general-purpose models.
- Numeric Expression Tokenization: Building on the insight that check-worthy claims often contain statistics or quantities, they implemented a "Segment2Token" process. This replaced digits and numerical expressions (like percentages or currency) with a generic
<number>token, helping the model generalize better across different specific values. - Hashtag Consolidation: They used a "Segment2Root" method to group semantically similar hashtags (e.g., #COVID19, #coronavirus, #Corona-virus) under a single umbrella term to reduce vocabulary noise.
- Model Architecture: A three-layer CNN with 32 filters and varying window sizes (2, 4, and 7) to act as feature extractors, followed by a sigmoid activation function to produce a ranking score.

Datasets
The team utilized the official CLEF 2020 CheckThat! dataset, which focused exclusively on COVID-19. To address the small size of the training set, they experimented with data augmentation using three external rumor-detection datasets: PHEME, Twitter 15, and Twitter 16.
Key Insights and Ideas
- Numbers Matter: The most significant performance boost came from the specialized treatment of numeric expressions. Check-worthy claims are frequently quantitative ("10% of patients need ICU care"), and tokenizing these allows the model to recognize the structure of a claim rather than getting hung up on specific, ever-changing statistics.
- Domain Expertise is Crucial: Using CT-BERT (pre-trained on COVID tweets) provided a clear advantage over standard uncased BERT. This highlights the importance of using language models that understand the specific jargon and context of a crisis.
- The Augmentation Paradox: While data augmentation is usually helpful, the authors found that adding general rumors from other topics (like the Charlie Hebdo shooting) didn't always improve performance. The "check-worthiness" of a tweet is highly subjective and topic-dependent, making it difficult to transfer knowledge from one type of crisis to another.
Key Results
The team submitted three models for the final evaluation. Their best-performing system (Model 2) utilized CT-BERT with numeric tokenization and achieved:
- Mean Average Precision (MAP): 0.78 on the test set.
- Precision@10: 1.00 (meaning all top 10 ranked tweets were indeed check-worthy).
- Overall Rank: 4th place in the competition.
This model significantly outperformed the competition's baselines, including a Random Classification (MAP = 0.35) and an SVM with N-gram prediction (MAP = 0.69).
Limitations and Future Work
The authors noted several challenges:
- Dataset Size: The primary COVID-19 dataset was relatively small (672 training tweets), limiting the complexity of the models they could reliably train.
- Subjectivity: Check-worthiness is inherently subjective; different fact-checking organizations might prioritize different types of claims based on their audience or mission.
- Future Directions: The team suggests that future work should incorporate "pragmatic information," such as the author's profile and social media behavior, to better simulate how actual users (and fact-checkers) evaluate the importance of a claim in the wild.
Summary updated