iCompass at NLP4IF-2021–Fighting the COVID-19 Infodemic

Wassim Henia, Oumayma Rjab, Hatem Haddad, Chayma FouratiNAACL 2021

TL;DR

This paper tackles the critical challenge of combating the COVID-19 infodemic by developing a system for detecting harmful and false information in Arabic social media. Leveraging a suite of pre-trained Arabic language models, the iCompass team fine-tuned them to classify tweets based on various criteria, ultimately achieving a strong F1 score with a BERT-base-arabic model.

What Problem Does This Paper Address?

The COVID-19 pandemic brought about an unprecedented surge in social media activity, but also a dangerous rise in misinformation and "infodemics." This paper focuses on the specific challenge of identifying and combating false information related to COVID-19 within the Arabic-speaking online community. The authors highlight that while social media facilitates information exchange, it also serves as a breeding ground for harmful content, which can have severe consequences, especially during crises. The NLP4IF (Natural Language Processing for Internet Freedom) task aimed to address this by developing systems capable of predicting various properties of COVID-19 related tweets, such as their harmfulness, verifiability, and potential for spreading misinformation.

Methods

The iCompass team employed a multi-label classification approach using several state-of-the-art pre-trained Arabic language models. These models were fine-tuned on a specific dataset for the task. The models utilized include:

  • ALBERT Arabic: A lighter version of BERT designed for efficiency.
  • Arabic BERT: A BERT model specifically trained on Arabic text.
  • AraBERT: Another BERT variant trained on a large corpus of Arabic news.
  • ARBERT: A BERT model focused on Modern Standard Arabic (MSA).
  • MARBERT: A BERT model specifically trained on a large dataset of Arabic tweets, designed to capture dialectal variations.

The fine-tuning process involved adding an untrained layer of neurons on top of the pre-trained models and then adjusting the weights of the last layers to adapt them to the specific task of infodemic detection. The researchers experimented with various hyperparameters to optimize performance.

Datasets

The primary dataset used for this work was provided by the NLP4IF-2021 competition. This dataset consists of:

  • Training dataset: 2536 tweets.
  • Development dataset: 520 tweets.

These tweets were labeled based on seven questions, including whether a tweet contains a verifiable factual claim, the extent of false information, its interest to the general public, its harmfulness, and whether it requires verification or attention. The tweets were in Modern Standard Arabic (MSA), and the data underwent preprocessing to remove emojis, URLs, punctuation, duplicated characters, diacritics, and non-Arabic words.

Key Insights and Ideas

  • The importance of dialectal Arabic: The authors note that models trained on a broader range of Arabic, including dialects (like MARBERT and Arabic BERT), can potentially perform better on social media data, which often contains informal language.
  • Fine-tuning is effective for specialized tasks: Even with relatively small annotated datasets, fine-tuning large pre-trained language models can yield significant performance improvements.
  • BERT-base-arabic's surprising strength: Despite the availability of more specialized Arabic models, the standard BERT-base-arabic model, when fine-tuned, achieved the best overall performance on the test set. This suggests that its pre-training data, which may include non-MSA content, was beneficial for this specific task.

Key Results

The iCompass system achieved a commendable F1 score of 0.748 on the test set, ranking second overall in the NLP4IF-2021 competition. Among the individual models tested, BERT-base-arabic demonstrated the highest performance on the development dataset with an F1 score of 0.7827. The paper also provides detailed results for each of the seven classification questions, showing varying levels of performance across different aspects of infodemic detection. For instance, the model performed best on questions related to general interest (Q3: 0.881) and harmfulness to society (Q6: 0.885), while performance was lower on the need for verification (Q5: 0.544).

Limitations and Future Work

The authors acknowledge that while their BERT-base-arabic model performed well, there is still room for improvement. Future work suggested includes:

  • Developing larger contextualized pre-trained models: Exploring even more powerful language models could lead to better performance.
  • Improving the current COVID-19 Infodemic Detection: Further research into more sophisticated techniques for identifying misinformation and harmful content is encouraged.

The paper also implicitly highlights the challenge of handling the nuances and rapid evolution of online discourse, especially in the context of a global crisis.

Summary updated

BibTeX
@inproceedings{henia2021icompass,
  author = {Wassim Henia and Oumayma Rjab and Hatem Haddad and Chayma Fourati},
  title = {{iCompass at NLP4IF-2021–Fighting the COVID-19 Infodemic}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure