AAST-NLP at ClimateActivism 2024: Ensemble-Based Climate Activism Stance and Hate Speech Detection : Leveraging Pretrained Language Models

Ahmed El-Sayed, Omar NasrEACL 2024

TL;DR

This paper tackles the growing problem of hate speech and divisive language targeting climate activists on social media. The authors developed an ensemble-based approach using pre-trained language models to detect hate speech, identify targets of such speech, and classify the stance of tweets related to climate activism, achieving competitive results across three subtasks.

What Problem Does This Paper Address?

Climate activism has become a powerful force for environmental advocacy, but social media platforms, while enabling mobilization, have also become breeding grounds for hate speech and harassment directed at these activists. This online negativity can undermine constructive efforts and discourage participation. The Climate Activism 2024 shared task aims to address this by providing a dataset and encouraging research into detecting hate speech and stance related to climate activism. This paper specifically focuses on building robust models for these tasks, acknowledging the challenges posed by imbalanced datasets and the need for accurate identification of harmful content.

Methods

The core of the authors' approach relies on ensemble learning applied to pre-trained language models (PLMs). They experimented with several powerful PLMs, including RoBERTa, XLM-RoBERTa, and HateBERT, fine-tuning them on the provided datasets.

For Subtask A (Hate Speech Detection) and Subtask C (Stance Detection), they employed an ensemble of the top-performing models, using a hard voting mechanism to determine the final prediction.

For Subtask B (Targets of Hate Speech Identification), they introduced an interesting hybrid approach. They combined Named Entity Recognition (NER) modules (SpaCy and a BERT-based NER) to extract key entities like "Person," "Organization," and "Community." These extracted features were then used in conjunction with the PLMs for classification.

To address the challenge of class imbalance in the datasets, they experimented with resampling techniques, specifically Random Under-sampling (RUS) and Random Over-sampling (ROS), ultimately finding ROS to be more effective. They also explored various loss functions, with Dice Loss proving to be a beneficial choice for NLP tasks with imbalanced data.

Datasets

The research utilizes the dataset provided for the Climate Activism Stance and Hate Event Detection at CASE 2024 shared task. This dataset comprises tweets related to climate activism and is divided into three subtasks:

  • Subtask A: Hate Speech Detection: A binary classification task (Hate Speech vs. No Hate Speech).
  • Subtask B: Targets of Hate Speech Identification: A multiclass classification task (Individual, Organization, Community).
  • Subtask C: Stance Detection: A multiclass classification task (Support, Oppose, Neutral).

The paper highlights the presence of class imbalances within these datasets, particularly for minority classes, which posed a significant challenge.

Key Insights and Ideas

  • Ensembling is Key: Combining multiple pre-trained language models, even those with slightly different strengths (e.g., one excelling in recall, another in precision), can lead to superior overall performance compared to individual models.
  • NER for Target Identification: Integrating NER for identifying specific targets of hate speech (individuals, organizations) proved to be a valuable strategy, especially when combined with powerful language models.
  • Dice Loss for Imbalance: Dice Loss, a loss function tailored for imbalanced data, showed promise in improving model performance on challenging classification tasks.
  • ROS Outperforms RUS: For addressing class imbalance, Random Over-sampling (ROS) was found to be more effective than Random Under-sampling (RUS) in this context.
  • The Double-Edged Sword of Social Media: The paper underscores how platforms that empower climate activists also facilitate the spread of harmful content against them, highlighting the urgent need for effective detection mechanisms.

Key Results

The authors' ensemble-based approach achieved competitive results across the three subtasks:

  • Subtask A (Hate Speech Detection): Their ensemble model secured the second place among 23 participating teams. The Top-5 Ensemble achieved an F1-score of 0.8914.
  • Subtask B (Targets of Hate Speech Identification): Their ensemble model also secured the second position among 18 participating teams. The Top-5 Ensemble achieved an F1-score of 0.7665.
  • Subtask C (Stance Detection): Their model achieved the fifth place. The Top-3 Ensemble achieved an F1-score of 0.7398.

Notably, RoBERTa often showed strong individual performance, particularly in precision for Subtask A, while HateBERT exhibited high recall for the same subtask, reinforcing the benefit of ensembling.

Limitations and Future Work

The authors acknowledge that while their ensemble approach showed improvement, the marginal gains in Subtask C (Stance Detection) raise questions about the computational cost versus the benefit for real-time implementation.

Future work suggested includes:

  • Exploring more advanced language models: Incorporating models like the FALCON series or Mistral could further enhance performance.
  • Synthetic data generation: Creating synthetic data to improve model robustness and performance on underrepresented classes.
  • Hyperparameter tuning: Investigating different hyperparameter configurations to optimize model performance.
  • Refining ensemble strategies: Further exploration of ensemble techniques to maximize gains.

The paper concludes that with further refinement, this approach can significantly contribute to mitigating the issue of hate speech against climate activists.

Summary updated

BibTeX
@inproceedings{elsayed2024aast,
  author = {Ahmed El-Sayed and Omar Nasr},
  title = {{AAST-NLP at ClimateActivism 2024: Ensemble-Based Climate Activism Stance and Hate Speech Detection : Leveraging Pretrained Language Models}},
  booktitle = {EACL},
  year = {2024},
}
Figure