SussexAI at ArAIEval Shared Task: Mitigating Class Imbalance in Arabic Propaganda Detection

Mary Fouad, Julie WeedsACL 2024

TL;DR

This paper tackles the challenge of detecting propaganda techniques in Arabic text, a task complicated by severe class imbalance. The authors propose using the AraBERT model and explore data augmentation and random truncation techniques to mitigate this imbalance, ultimately advocating for the use of macro-F1 scores for more robust evaluation.

What Problem Does This Paper Address?

Propaganda detection is crucial for combating misinformation and manipulation, especially in today's polarized world. While research has focused on English text, there's a growing need for similar capabilities in other languages, like Arabic. The ArAIEval Shared Task aims to address this by identifying propaganda techniques and their exact spans within Arabic text. A significant challenge in this task, as highlighted by the authors, is the extreme class imbalance within the dataset. This means that some propaganda techniques appear very rarely, making it difficult for machine learning models to learn to detect them effectively. Standard evaluation metrics like micro-F1 can be misleading in such scenarios, as they can be heavily influenced by the performance on majority classes.

Methods

The core of the proposed solution is the AraBERT model, a BERT-based language model pre-trained on Arabic text, fine-tuned for a sequence tagging task. This means the model learns to assign a label (indicating a propaganda technique or "no propaganda") to each token in a sentence.

To combat the class imbalance, two primary techniques were explored:

  • Data Augmentation: This involved generating synthetic data for minority classes. Specifically, the authors used the AraBERT Masked Language Model (MLM) to predict masked words within propaganda snippets. By masking individual words in propaganda spans and using AraBERT's fill-mask function, they generated new examples. The goal was to increase the representation of underrepresented propaganda techniques.
  • Random Truncation: This technique was applied to reduce the dominance of "no-propaganda" tokens, which constitute the vast majority of tokens in the dataset. Random truncation involved removing tokens outside the identified propaganda spans, with a small number of surrounding tokens retained. This aims to rebalance the dataset by reducing the overwhelming number of negative examples.

Datasets

The primary dataset used in this work is the ArAIEval24 dataset. This dataset was provided in JSON format and contained annotated instances of propaganda techniques and their corresponding spans within Arabic text. The authors processed this dataset for training, development, and evaluation.

Key Insights and Ideas

  • Class Imbalance is a Major Hurdle: The paper strongly emphasizes that class imbalance is a critical issue in Arabic propaganda detection, significantly impacting model performance and evaluation.
  • Micro-F1 Can Be Deceptive: The authors argue that micro-F1 scores can be misleading for imbalanced datasets, potentially favoring models that simply predict the majority class.
  • Macro-F1 Offers a Balanced View: Macro-F1 is presented as a more reliable metric because it averages F1 scores across all classes, giving equal weight to each, regardless of its frequency. This provides a better understanding of a model's ability to detect minority classes.
  • Random Truncation Shows Promise: While data augmentation had a negative impact, random truncation demonstrated a significant improvement in model performance, particularly in predicting minority classes.

Key Results

The paper reports the following F1 scores on the test dataset:

  • AraBERT Basic: Macro F1 = 0.05, Micro F1 = 0.06
  • AraBERT Augmented: Macro F1 = 0.03, Micro F1 = 0.05 (Performance decreased with augmentation)
  • AraBERT Rand-Truncated: Macro F1 = 0.10, Micro F1 = 0.12

The random truncation method significantly improved the macro-F1 score from 0.05 to 0.10, indicating better performance across all classes, including minority ones. The basic AraBERT model failed to predict more than 12 minority classes, a number that improved with random truncation.

Limitations and Future Work

  • Data Augmentation's Negative Impact: The authors note that their data augmentation strategy, surprisingly, worsened the model's performance. They attribute this to the potential for the augmented data to exacerbate the existing class imbalance.
  • Future Work - Active Learning: The authors suggest exploring active learning strategies to discover and sample more instances of minority classes. This could further enhance class balance and lead to better minority class classifiers.
  • Future Work - Bootstrapping: They also propose using a more effective minority class classifier to curate examples from large unannotated corpora, ultimately bootstrapping a better overall classifier.

Summary updated

BibTeX
@inproceedings{fouad2024sussexai,
  author = {Mary Fouad and Julie Weeds},
  title = {{SussexAI at ArAIEval Shared Task: Mitigating Class Imbalance in Arabic Propaganda Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure