CMTA: COVID-19 Misinformation Multilingual Analysis on Twitter

Raj Pranesh, Mehrdad Farokhenajd, Ambesh Shekhar, Genoveva Vargas-SolarACL 2021

TL;DR

This paper introduces CMTA, a novel multilingual BERT-based model designed to detect and classify COVID-19 misinformation on Twitter across various languages. By leveraging transfer learning, CMTA achieves performance comparable to specialized monolingual models, offering a robust solution for understanding the global spread of health-related falsehoods.

What Problem Does This Paper Address?

The COVID-19 pandemic has been accompanied by a significant surge in misinformation and disinformation across social media platforms. This "infodemic" poses a serious threat to public health by sowing confusion, promoting harmful practices, and undermining trust in credible sources. While many studies have focused on English-language misinformation, the global nature of the pandemic necessitates tools that can analyze and combat false narratives across multiple languages. Twitter, being a primary platform for real-time news and opinion sharing, is a critical area for this analysis. The challenge lies in developing a unified approach that can effectively process and classify misinformation in diverse linguistic contexts without requiring separate models for each language.

Methods

The core of the CMTA system is a multilingual BERT (mBERT) model. BERT, a powerful transformer-based language model, excels at understanding contextual relationships within text. mBERT extends this capability to over 100 languages. The authors fine-tuned mBERT for the specific task of misinformation detection.

The CMTA architecture involves several key stages:

  • Tokenization: Using mBERT's multilingual tokenizer to convert text into numerical representations that the model can process.
  • Feature Extraction: Exploiting the hidden states generated by mBERT's layers to capture rich contextual embeddings of words and sentences. They specifically focus on combining vectors from multiple layers to achieve a comprehensive representation.
  • Linear Convolution and Pooling: Applying 1D convolutional layers to the extracted features to identify correlations among tokens, followed by global average pooling to derive a single sentence representation.
  • Classification: A final linear layer with a Softmax activation function categorizes the input text into one of three classes: 'false', 'partly false', or 'misleading'.

The authors emphasize that this approach allows for a single, unified framework to handle multilingual data, leveraging the power of transfer learning from mBERT.

Datasets

The study utilized two primary sources for its dataset:

  • Poynter Institute's "CoronaVirusFacts/DatosCoronaVirus Alliance Database": This database provided manually verified and annotated social media information, news, posts, and articles related to COVID-19. The authors extracted misinformation data in both English and Spanish from this source.
  • Alam et al. (2020) fact-checked tweet dataset: This dataset, available at a public repository, provided labeled tweets in English and Arabic. The authors specifically used the false-labeled tweets from this source.

In total, they compiled 9,502 micro-articles across 9 misinformation classes. For training, they merged minority classes ('No evidence', 'Four Pinocchios', 'Incorrect', 'Three Pinocchios', 'Two Pinocchios', and 'Mostly False') into the majority classes ('False', 'Partially false', and 'Misleading') to create a more balanced dataset.

For inference and analysis, they collected a large dataset of approximately 2.1 million multilingual tweets spanning eight major languages: English, Spanish, Indonesian, French, Japanese, Thai, Hindi, and German. This dataset was derived from an existing dataset of COVID-19 related tweets.

Key Insights and Ideas

  • Transfer Learning for Multilingual Misinformation: The study strongly suggests that transfer learning via a multilingual model like mBERT is a reliable and robust method for analyzing misinformation across different languages, performing comparably to specialized monolingual models.
  • The "Infodemic" is Global: The analysis highlights that misinformation is not confined to a single language but is a widespread phenomenon across multiple linguistic communities.
  • Classifying Nuance: The authors acknowledge the difficulty in distinguishing between misinformation and disinformation based on intent. Their classification into 'false', 'partly false', and 'misleading' attempts to capture the spectrum of inaccurate information.
  • Hashtags as Valuable Information: The decision to retain hashtags during preprocessing, unlike other special characters, is based on the idea that they can carry significant contextual information relevant to misinformation.

Key Results

  • CMTA Performance: The proposed CMTA model achieved an accuracy of 82.17% and an F1-score of 82.54% on the test dataset. Precision was reported at 82.07% and recall at 82.30%.
  • Monolingual vs. Multilingual: The CMTA model's performance was found to be equivalent to or even surpass several monolingual BERT models when tested on their respective languages. For instance, EnglishBERT scored an F1-score of 77.9%, while CMTA achieved 77.79% on English tweets, demonstrating its strong generalization capabilities.
  • Language-wise Distribution: The analysis revealed varying patterns of misinformation across languages. Spanish tweets showed a high frequency of 'False' labels, while German tweets had a surge in 'Misleading' content. Japanese tweets exhibited a significant proportion of 'Misleading' content (around 50%).
  • Temporal Trends: The study observed month-wise distributions of misinformation types, noting fluctuations and surges in specific categories (e.g., 'Misleading' tweets in February for German).

Limitations and Future Work

  • Generalizability: The authors acknowledge that the analysis was performed on a "limited dataset," and therefore, the results cannot be fully generalized.
  • Intent Determination: The paper notes the practical difficulty of computationally determining the intent behind misinformation (i.e., distinguishing misinformation from disinformation).
  • Future Work: The authors plan to collect more annotated training data to expand the multilingual dataset and improve the model's robustness and contextual understanding. They aim to enhance performance in the classification task and contribute to building more reliable social media platforms.

Summary updated

BibTeX
@inproceedings{pranesh2021cmta,
  author = {Raj Pranesh and Mehrdad Farokhenajd and Ambesh Shekhar and Genoveva Vargas-Solar},
  title = {{CMTA: COVID-19 Misinformation Multilingual Analysis on Twitter}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure