CEIA-NLP at CASE 2022 Task 1: Protest News Detection for Portuguese

Diogo Fernandes Costa Silva, Adalberto Ferreira Barbosa Junior, Gabriel Marques, Anderson da Silva Soares, Arlindo Rodrigues Galvão FilhoCASE @ EMNLP 2022 · OpenReview

TL;DR

This paper tackles the challenge of detecting protest-related news in Portuguese, a low-resource language. Their winning solution leveraged multilingual transformer models, demonstrating that training on data from related languages significantly boosts performance compared to monolingual models when data is scarce.

What Problem Does This Paper Address?

The proliferation of social media and online platforms has led to an explosion of information, making it difficult to track and analyze social movements and political events. Specifically, identifying and classifying news related to protests is crucial for understanding societal dynamics and informing policy decisions. However, for many languages, including Portuguese, there's a scarcity of labeled data for training effective natural language processing (NLP) models. This paper addresses this "low-resource" challenge by investigating how well monolingual and multilingual transformer models perform on Portuguese protest news detection.

Methods

The authors employed transformer-based models, a standard in modern NLP, for document classification. They experimented with both monolingual (Portuguese-specific) and multilingual models. The key idea was to see if leveraging data from other languages (English and Spanish, in this case) within a multilingual framework could compensate for the limited Portuguese protest news data. They fine-tuned pre-trained models like BERT, mBERT, RoBERTa, and xlm-RoBERTa. A crucial aspect of their methodology was exploring the impact of sequence length on model performance, hypothesizing that longer sequences might capture more context relevant to protest events. Hyperparameter tuning was performed using a grid search on a development set.

Datasets

The primary dataset used was provided by the CASE 2022 shared task, specifically for the multilingual protest news detection subtask. This dataset comprises socio-political events in text and includes three languages: English, Spanish, and Portuguese. The authors split this dataset into training (80%) and validation (20%) sets. They also mention that the dataset is a combination of data from CASE 2021 and new test data for the document classification subtask.

Key Insights and Ideas

  • Multilingualism Wins in Low-Resource Settings: The most significant finding is that multilingual models consistently outperform their monolingual counterparts when dealing with limited data for a specific language (Portuguese). This highlights the power of transfer learning across languages for tasks where one language has fewer examples.
  • Longer Sequences, Better Performance: Increasing the maximum sequence length fed into the transformer models led to improved performance across all tested models. This suggests that capturing more of the document's context is vital for accurately identifying protest-related content.
  • The Power of xlm-RoBERTa: The xlm-RoBERTa model, a multilingual variant of RoBERTa, emerged as the top performer, indicating its strong capacity for cross-lingual understanding and adaptation.

Key Results

The proposed xlm-RoBERTa model achieved the 1st place in the Portuguese document classification subtask of the CASE 2022 competition, securing a Macro F1 score of 0.8007 on the test set. On the validation set, increasing the sequence length to 512 tokens for xlm-RoBERTa resulted in the highest Macro F1 score of 0.8818.

Limitations and Future Work

The authors acknowledge that while multilingual models excel in low-resource scenarios, monolingual models might perform better in high-resource settings. They also note that the size of the maximum sequence length significantly impacts performance, and larger transformer models require substantial GPU resources for processing long texts. Future work could explore other methodologies like ensemble approaches, data augmentation, and few-shot learning techniques. Additionally, they suggest investigating architectures that can handle much longer input sequences. The paper also touches upon the ethical implications of training models on vast, potentially biased, unannotated data, suggesting future work should focus on measuring and mitigating such biases.

Summary updated

Abstract

This paper summarizes our work on the document classification subtask of Multilingual protest news detection of the CASE @ ACL-IJCNLP 2022 workshok. In this context, we investigate the performance of monolingual and multilingual transformer-based models in low data resources, taking Portuguese as an example and evaluating language models on document classification. Our approach became the winning solution in Portuguese document classification achieving 0.8007 F1 Score on Test set. The experimental results demonstrate that multilingual models achieve best results in scenarios with few dataset samples of specific language, because we can train models using datasets from other languages of the same task and domain.
BibTeX
@inproceedings{silva2022ceia,
  author = {Diogo Fernandes Costa Silva and Adalberto Ferreira Barbosa Junior and Gabriel Marques and Anderson da Silva Soares and Arlindo Rodrigues Galvão Filho},
  title = {{CEIA-NLP at CASE 2022 Task 1: Protest News Detection for Portuguese}},
  booktitle = {CASE @ EMNLP},
  year = {2022},
  url = {https://openreview.net/forum?id=N9hTfz8a2o},
}
Figure