Detecting Fake News Conspiracies with Multitask and Prompt-Based Learning

Cheikh Brahim El Vaigh, Thomas Girault, Cyrielle Mallart, Duc Hau NguyenMediaEval 2021 · OpenReview

TL;DR

This paper tackles the challenge of detecting coronavirus-related conspiracy theories on Twitter by framing it as a multi-label classification problem. They found that a multitask learning approach, where models learn all tasks simultaneously, significantly outperforms individual task models, highlighting the benefits of shared learning for related NLP tasks.

What Problem Does This Paper Address?

The rapid spread of misinformation, particularly concerning the COVID-19 pandemic, on social media platforms like Twitter poses a significant challenge. Unlike traditional media, social media lacks rigorous fact-checking, leading to a proliferation of conspiracy theories. This paper addresses the need for automated systems to identify and classify these conspiracy theories, specifically focusing on those related to coronavirus. The task is broken down into three subtasks: identifying if a tweet relates to a conspiracy, what specific conspiracy it mentions, and a combined classification.

Methods

The authors explored several approaches using transformer-based models. They first experimented with separated multilabel models, fine-tuning individual BERT-based models (including BERTweet, a model pre-trained on tweets, and toxic-bert, trained on toxic language) for each of the three tasks. They also investigated a prompt-based learning approach for Task 1, treating tweet classification as a text generation problem using DistilGPT-2. The core innovation lies in their multitask learning model, which concatenates the labels from all three tasks and trains a single model to perform them jointly. This approach leverages the inherent relationships between the tasks, aiming for improved performance by learning shared representations.

Datasets

The study utilized datasets provided by the MediaEval-2021 Fake News detection challenge. These datasets were specifically curated for the three subtasks related to coronavirus conspiracy theories on Twitter. The paper does not introduce new datasets but rather uses existing ones for evaluation.

Key Insights and Ideas

  • Multitask learning reigns supreme: The most significant finding is that training a single model on all three tasks simultaneously (multitask learning) yields superior results compared to training separate models for each task. This suggests that the tasks are sufficiently related to benefit from shared learning.
  • Pre-training matters, but context is key: While models pre-trained on general language (like BERT) or even toxic language showed some promise, the BERTweet model, pre-trained specifically on tweet data, performed particularly well. This underscores the importance of domain-specific pre-training for social media text.
  • Prompt-based learning shows potential but needs more resources: The prompt-based approach, while not outperforming traditional fine-tuning on Task 1 in this study, is identified as a promising avenue, especially for tasks with limited labeled data. The authors believe it could be more effective in a multitask setting.
  • Smaller models can be surprisingly effective: Models like bert-small and bert-tiny achieved competitive results, suggesting that efficient architectures can still be powerful for this task.

Key Results

The multitask learning model outperformed individual models on all three tasks, achieving the best overall scores. For instance, on the test set, the multitask model achieved an MCC score of 0.73 for Task 1, 0.73 for Task 2, and 0.68 for Task 3, compared to the separated task models which achieved 0.60, 0.72, and 0.66 respectively. The BERTweet model, when fine-tuned individually for Task 1, achieved an MCC of 0.58, while the prompt-based DistilGPT-2 model achieved 0.45.

Limitations and Future Work

The authors acknowledge that the official MCC scores, while good, indicate room for improvement, particularly for Task 1. They also note that the prompt-based model was only trained on Task 1 due to time and resource constraints, and its full potential might be realized when extended to multitask learning. Future work includes exploring larger generative models for prompt-based training, applying the prompt model to all three tasks, and developing a multitask prompt-based model to combine their most promising approaches. They also mention an unsuccessful attempt to generate fake training samples using GPT-2 due to annotation limitations.

Summary updated

BibTeX
@inproceedings{vaigh2021detecting,
  author = {Cheikh Brahim El Vaigh and Thomas Girault and Cyrielle Mallart and Duc Hau Nguyen},
  title = {{Detecting Fake News Conspiracies with Multitask and Prompt-Based Learning}},
  booktitle = {MediaEval},
  year = {2021},
  url = {https://openreview.net/forum?id=Ad_xC0tEnB},
}
Figure