CamPros at CASE 2022 Task 1: Transformer-based Multilingual Protest News Detection

Kumari Neha, Mrinal Anand, Tushar Mohan, Ponnurangam Kumaraguru, Arun Balaji BuduruCASE @ EMNLP 2022 · DOI

TL;DR

This paper tackles the challenge of automatically detecting socio-political protests from news articles across multiple languages, especially in low-resource settings where training data is scarce. They propose a multilingual transformer-based approach, specifically leveraging XLM-Roberta, and achieve strong performance, securing second place in the Hindi zero-shot event detection task.

What Problem Does This Paper Address?

Automatically identifying socio-political protests and crisis events from the vast amount of information generated online is crucial for timely interventions and understanding public sentiment. However, a significant hurdle is the scarcity of labeled training data, particularly for low-resource languages. This makes it difficult to train data-hungry deep learning models effectively. The paper addresses this by focusing on multilingual and zero-shot learning approaches for document-level event detection, aiming to build models that can generalize across languages with limited or no specific training data.

Methods

The core of the proposed method relies on pre-trained multilingual transformer models. Specifically, the authors fine-tune XLM-Roberta, a model pre-trained on a massive corpus of 100 languages, for the task of document-level event detection. They frame the problem as a sequence classification task and experiment with different classification layers on top of the transformer embeddings. Their preliminary analysis indicated that a sigmoid classification layer performed best, leading to their final proposed model: XLM-Roberta with a sigmoid layer. They also experimented with mBERT and LASER as baselines. For data processing, they maintained language agnosticism, avoiding language-specific preprocessing and removing URLs and repeated symbols.

Datasets

The research utilizes datasets provided for the CASE 2022 Task 1: Multilingual Protest News Detection Shared Task.

  • Training Data: Consists of documents from English, Spanish, and Portuguese. The training data is noted to be highly imbalanced.
  • Testing Data: Includes documents from English, Spanish, Portuguese, Hindi, Mandarin, Turkish, and Urdu. Notably, no training data was provided for Hindi, Mandarin, Turkish, and Urdu, necessitating a zero-shot learning approach for these languages.

Key Insights and Ideas

  • XLM-Roberta's strength in multilingual settings: The paper demonstrates that XLM-Roberta, with its extensive multilingual pre-training, is a powerful foundation for cross-lingual event detection, even in zero-shot scenarios.
  • Sigmoid layer's effectiveness: The choice of a sigmoid classification layer on top of XLM-Roberta proved to be a key factor in achieving competitive results, outperforming other configurations.
  • Zero-shot learning is viable for low-resource languages: The success in Hindi and the performance on other zero-shot languages highlight the potential of transfer learning from high-resource to low-resource languages for this task.
  • Data imbalance is a persistent challenge: While the authors experimented with oversampling and undersampling, these techniques did not yield significant improvements, suggesting that more advanced methods might be needed to fully address data imbalance.

Key Results

The system achieved an average F1 score of 0.73 for document-level event detection across all languages.

  • Hindi: The proposed XLM-Roberta+Sigmoid model secured 2nd position in the zero-shot Hindi document-level event detection task with an F1 score of 0.80.
  • Spanish: The system achieved a 4th position with an F1 score of 0.69.
  • English: The best model for English was XLM-Roberta+Sigmoid with a macro-F1 score of 0.77.
  • Portuguese: The proposed framework also outperformed other baselines for Portuguese, achieving a macro-F1 score of 0.76.

Limitations and Future Work

  • Token Limit: The maximum token size for XLM-Roberta (512 tokens) is a limitation for processing longer documents, potentially leading to missed information. The authors used a 256-token limit in their final approach.
  • Data Imbalance: While addressed, the impact of data imbalance on model performance was not fully mitigated, as oversampling and undersampling did not lead to significant improvements.
  • Comparison with Multilingual BERT: For Spanish and Mandarin, the multilingual BERT with a softmax layer performed better than their proposed XLM-Roberta approach, indicating that different models might be optimal for different languages or tasks.

The authors intend to make their code publicly available, encouraging further research and community use. Future work could explore more advanced techniques for handling data imbalance and investigate why mBERT+Softmax performed better for Spanish and Mandarin.

Summary updated

BibTeX
@inproceedings{neha2022campros,
  author = {Kumari Neha and Mrinal Anand and Tushar Mohan and Ponnurangam Kumaraguru and Arun Balaji Buduru},
  title = {{CamPros at CASE 2022 Task 1: Transformer-based Multilingual Protest News Detection}},
  booktitle = {CASE @ EMNLP},
  year = {2022},
  doi = {10.18653/v1/2022.case-1.24},
}
Figure