QMUL-SDS at CheckThat! 2020: Determining COVID-19 Tweet Check-Worthiness Using an Enhanced CT-BERT with Numeric Expressions

Rabab Alkhalifa, Theodore Yoong, Elena Kochkina, Arkaitz Zubiaga, Maria LiakataarXiv 2020 · arXiv

TL;DR

Researchers from QMUL and partner institutions developed a system to automatically rank COVID-19 tweets by their "check-worthiness"—essentially identifying which claims are most important for fact-checkers to investigate. By combining a domain-specific language model (CT-BERT) with a convolutional neural network (CNN) and specialized tokenization for numeric expressions, their system achieved 4th place in the CLEF 2020 CheckThat! competition.

What Problem Does This Paper Address?

During the COVID-19 pandemic, social media became a primary source of information, but also a breeding ground for harmful misinformation and conspiracy theories. Fact-checking organizations are often overwhelmed by the sheer volume of posts.

The challenge isn't just identifying if a tweet is true or false, but determining its check-worthiness: Does it contain a claim that is of interest to a large audience and potentially harmful if left unverified? This paper tackles the task of automatically prioritizing these high-stakes tweets so that human fact-checkers can focus their limited resources where they matter most.

Methods

The authors proposed a deep learning pipeline centered around a 1D Convolutional Neural Network (CNN) that processes word-level vector representations.

Key components of their approach include:

  • Domain-Specific Embeddings: They compared standard BERT and ELMo against COVID-Twitter-BERT (CT-BERT), a model pre-trained specifically on COVID-19 related tweets. CT-BERT significantly outperformed general-purpose models.
  • Numeric Expression Tokenization: Building on the insight that check-worthy claims often contain statistics or quantities, they implemented a "Segment2Token" process. This replaced digits and numerical expressions (like percentages or currency) with a generic <number> token, helping the model generalize better across different specific values.
  • Hashtag Consolidation: They used a "Segment2Root" method to group semantically similar hashtags (e.g., #COVID19, #coronavirus, #Corona-virus) under a single umbrella term to reduce vocabulary noise.
  • Model Architecture: A three-layer CNN with 32 filters and varying window sizes (2, 4, and 7) to act as feature extractors, followed by a sigmoid activation function to produce a ranking score.

![Fig.

  1. General Model Architecture.](/figures/art_67b2c61446f94b1f9531185d1c3b970a)

Datasets

The team utilized the official CLEF 2020 CheckThat! dataset, which focused exclusively on COVID-19. To address the small size of the training set, they experimented with data augmentation using three external rumor-detection datasets: PHEME, Twitter 15, and Twitter 16.

Table 1. Number of posts and class distribution in the datasets used

Key Insights and Ideas

  • Numbers Matter: The most significant performance boost came from the specialized treatment of numeric expressions. Check-worthy claims are frequently quantitative ("10% of patients need ICU care"), and tokenizing these allows the model to recognize the structure of a claim rather than getting hung up on specific, ever-changing statistics.
  • Domain Expertise is Crucial: Using CT-BERT (pre-trained on COVID tweets) provided a clear advantage over standard uncased BERT. This highlights the importance of using language models that understand the specific jargon and context of a crisis.
  • The Augmentation Paradox: While data augmentation is usually helpful, the authors found that adding general rumors from other topics (like the Charlie Hebdo shooting) didn't always improve performance. The "check-worthiness" of a tweet is highly subjective and topic-dependent, making it difficult to transfer knowledge from one type of crisis to another.

Key Results

The team submitted three models for the final evaluation. Their best-performing system (Model 2) utilized CT-BERT with numeric tokenization and achieved:

  • Mean Average Precision (MAP): 0.78 on the test set.
  • Precision@10: 1.00 (meaning all top 10 ranked tweets were indeed check-worthy).
  • Overall Rank: 4th place in the competition.

This model significantly outperformed the competition's baselines, including a Random Classification (MAP = 0.35) and an SVM with N-gram prediction (MAP = 0.69).

Limitations and Future Work

The authors noted several challenges:

  • Dataset Size: The primary COVID-19 dataset was relatively small (672 training tweets), limiting the complexity of the models they could reliably train.
  • Subjectivity: Check-worthiness is inherently subjective; different fact-checking organizations might prioritize different types of claims based on their audience or mission.
  • Future Directions: The team suggests that future work should incorporate "pragmatic information," such as the author's profile and social media behavior, to better simulate how actual users (and fact-checkers) evaluate the importance of a claim in the wild.

Summary updated

Abstract

This paper describes the participation of the QMUL-SDS team for Task 1 of the CLEF 2020 CheckThat! shared task. The purpose of this task is to determine the check-worthiness of tweets about COVID-19 to identify and prioritise tweets that need fact-checking. The overarching aim is to further support ongoing efforts to protect the public from fake news and help people find reliable information. We describe and analyse the results of our submissions. We show that a CNN using COVID-Twitter-BERT (CT-BERT) enhanced with numeric expressions can effectively boost performance from baseline results. We also show results of training data augmentation with rumours on other topics. Our best system ranked fourth in the task with encouraging outcomes showing potential for improved results in the future.
BibTeX
@misc{alkhalifa2020qmul,
  author = {Rabab Alkhalifa and Theodore Yoong and Elena Kochkina and Arkaitz Zubiaga and Maria Liakata},
  title = {{QMUL-SDS at CheckThat! 2020: Determining COVID-19 Tweet Check-Worthiness Using an Enhanced CT-BERT with Numeric Expressions}},
  year = {2020},
  eprint = {2008.13160},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.13160},
}
Figure