IIITT at CASE 2021 Task 1: Leveraging Pretrained Language Models for Multilingual Protest Detection

Pawan Kalyan, Duddukunta Reddy, Adeep Hande, Ruba Priyadharshini, Ratnasingam Sakuntharaj, Bharathi Raja ChakravarthiACL 2021

TL;DR

This paper tackles the challenge of automatically detecting protest events across multiple languages (English, Spanish, Portuguese) using pre-trained language models. By fine-tuning BERT, RoBERTa, and DistilBERT and employing a soft voting strategy, the authors achieved strong performance, highlighting the potential of these models for real-world event detection.

What Problem Does This Paper Address?

In an era of rapid information dissemination, especially through social media, timely detection and analysis of protests and social movements are crucial. These events, often triggered by significant societal issues like pandemics or political conflicts, can quickly gain momentum and influence public opinion. However, manually monitoring and analyzing vast amounts of multilingual text data for these events is a monumental task. This paper addresses the need for automated systems that can accurately identify sentences indicating protest events across different languages, moving beyond traditional news-based analysis to encompass the nuances of social media discourse.

Methods

The core of this work involves leveraging pre-trained multilingual transformer models for sentence classification. Specifically, the authors fine-tuned three models:

  • BERT (Bidirectional Encoder Representations from Transformers): A foundational model known for its deep bidirectional pre-training. They used the bert-base-multilingual-cased variant, trained on 104 languages.
  • DistilBERT: A distilled, smaller, faster, and lighter version of BERT, also multilingual.
  • RoBERTa (Robustly Optimized BERT Pretraining Approach): An optimized version of BERT that differs in its pre-training strategy, notably using dynamic masking and omitting the Next Sentence Prediction (NSP) task.

To handle the multilingual aspect, they combined the English, Spanish, and Portuguese datasets. After fine-tuning each model individually, they implemented a soft voting mechanism. In soft voting, each model predicts the probability of a sentence belonging to a class, and these probabilities are combined (with equal weights in this case) to produce a final prediction. This approach aims to leverage the strengths of each model and improve overall robustness. The fine-tuned models' embeddings were fed into an LSTM layer, followed by a global average pooling layer and fully connected layers for classification.

Datasets

The primary dataset used in this work is the one provided for the CASE 2021 Task 1: Multilingual Protest Detection. This dataset comprises 26,208 sentences across three languages: English, Spanish, and Portuguese. The sentences are labeled as either "Event" (indicating an event that has transpired) or "Not-event." The dataset exhibits class imbalance, with significantly more "Not-event" sentences than "Event" sentences, and a disproportionately large number of English samples compared to Spanish and Portuguese. The authors split this dataset into training and validation sets at an 80:20 ratio.

Key Insights and Ideas

  • Multilingual Models are Key: The success of using multilingual pre-trained models like BERT and DistilBERT demonstrates their effectiveness in capturing linguistic nuances across different languages for event detection.
  • Soft Voting Enhances Performance: Combining the predictions of multiple models through soft voting proved to be more effective than any single model, leading to improved macro F1-scores across all languages.
  • Class Imbalance is a Major Hurdle: The significant imbalance between "Event" and "Not-event" classes negatively impacted model performance, particularly for the minority "Event" class. This suggests that techniques to address imbalance are crucial for this task.
  • DistilBERT's Efficiency: DistilBERT, despite being a distilled model, achieved a better F1-score than RoBERTa, highlighting its competitive performance and efficiency.

Key Results

The soft voting approach achieved the following macro F1-Scores on the test set:

  • English: 0.8291
  • Spanish: 0.7578
  • Portuguese: 0.7951

On the validation set, the models showed varying performance:

  • mBERT: Macro F1-Score of 0.790
  • DistilBERT: Macro F1-Score of 0.810
  • RoBERTa: Macro F1-Score of 0.772
  • Soft Voting: Macro F1-Score of 0.827

The authors note that DistilBERT performed best among the individual models on the validation set.

Limitations and Future Work

The authors acknowledge the following limitations:

  • Class Imbalance: The dataset's imbalance between "Event" and "Not-event" classes significantly affected performance, with models performing better on the majority class.
  • Low Support for Spanish and Portuguese: The smaller number of training samples for Spanish and Portuguese compared to English might have influenced their performance.
  • Combined Corpus Impact: Combining datasets from different languages might have influenced the performance of the low-support datasets.

For future work, the authors intend to explore:

  • Class Weighting Techniques: To mitigate the impact of class imbalance.
  • Semi-supervised Approaches: To potentially leverage unlabeled data and improve performance.

Summary updated

BibTeX
@inproceedings{kalyan2021iiitt,
  author = {Pawan Kalyan and Duddukunta Reddy and Adeep Hande and Ruba Priyadharshini and Ratnasingam Sakuntharaj and Bharathi Raja Chakravarthi},
  title = {{IIITT at CASE 2021 Task 1: Leveraging Pretrained Language Models for Multilingual Protest Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure