IBM MNLP IE at CASE 2021 Task 1: Multigranular and Multilingual Event Detection on Protest News

Parul Awasthy, Jian Ni, Ken Barker, Radu FlorianACL 2021

TL;DR

This paper presents a robust system for detecting and classifying protest events in news articles across multiple languages and granularities. Leveraging the powerful XLM-R transformer model, their approach achieved top rankings in the CASE 2021 shared task, demonstrating the effectiveness of multilingual models for fine-grained event detection.

What Problem Does This Paper Address?

The paper tackles the challenge of Multilingual Protest News Detection, a crucial task for understanding global socio-political events. Existing event detection systems often struggle with multilingual data and varying levels of detail. This shared task specifically aimed to detect events at different granularities (document, sentence, and token levels) and across multiple languages (English, Hindi, Portuguese, and Spanish), including low-resource languages and zero-shot scenarios. The core problem is to build a unified system that can effectively process and extract event information from diverse news sources, overcoming language barriers and differing levels of specificity.

Methods

The authors' approach is built upon the XLM-R (Cross-lingual Language Model - RoBERTa), a powerful multilingual transformer-based language model. This model serves as the core text encoder, capable of understanding and processing text from various languages.

For the different subtasks, they employed a range of techniques:

  • Document and Sentence Classification (Subtasks 1 & 2): These were framed as binary text classification problems. They used XLM-R to generate sentence/document embeddings, which were then fed into a softmax classification layer. They experimented with both multilingual models (trained on data from all languages) and monolingual models (trained on data from a single language).
  • Event Sentence Coreference Identification (Subtask 3): This involved two main modules:
  • A classification module to determine if pairs of sentences are coreferent.
  • A clustering module that uses agglomerative clustering to group coreferent sentences into events.
  • Event Extraction (Subtask 4): This was treated as a sequence labeling task, using the IOB2 encoding scheme to identify event triggers and their arguments (time, facility, organizer, participant, place, target). They adapted a BERT-based baseline, using XLM-R for token-level contextualized embeddings.

A key aspect of their methodology was exploring different model configurations: multilingual, monolingual, and ensemble approaches, as well as zero-shot and weakly supervised methods for languages with limited data.

Datasets

The data for this task was derived from the method described in Hürriyetoğlu et al. (2021). The paper notes that while the task is multilingual, the data distribution is uneven, with significantly more data for English than for Portuguese and Spanish. Notably, there was no official training data provided for Hindi.

The authors created their own train/dev/test splits, with details summarized in Table 1. For languages with less data (Spanish and Portuguese), they sometimes used the data for development only or for training multilingual models. For Hindi, they utilized machine-translated data to create "weakly labeled" training data.

Key Insights and Ideas

  • Multilingual Models Excel for Fine-Grained Tasks: The paper highlights that for finer-grained tasks like sentence classification (Subtask 2), multilingual models often outperform language-specific monolingual models, especially for lower-resource languages. This suggests that pooling data across languages can significantly boost performance.
  • Monolingual Models Show Strong Zero-Shot Capabilities: Surprisingly, monolingual XLM-R models trained on English data achieved good zero-shot performance on Spanish and Portuguese development sets for document classification. This indicates the inherent cross-lingual transfer capabilities of these models.
  • Machine Translation for Low-Resource Languages is Viable: For Hindi, where no annotated data was available, using machine-translated data to create weakly labeled examples allowed them to train effective document classification models.
  • Ensemble Methods Provide Robustness: The use of ensemble systems (e.g., majority voting across multiple models) consistently improved performance and robustness across various subtasks.

Key Results

The authors' systems achieved impressive results, ranking highly across the shared task:

  • Overall Average F₁ Score: 81.2, which was noted as 2 F₁ points higher than the best score of other participants.
  • Rankings: Their submissions ranked 1st in nine out of thirteen tracks and 2nd in the remaining four tracks.
  • Subtask 1 (Document Classification): Monolingual models performed well, with multilingual models showing improvements when additional data was added.
  • Subtask 2 (Sentence Classification): Multilingual models outperformed monolingual ones for Portuguese and Spanish.
  • Subtask 3 (Event Coreference): The monolingual English-only model performed best across all languages, possibly due to the use of machine-translated data in the multilingual approach.
  • Subtask 4 (Event Extraction): The multilingual model showed better performance for Spanish and Portuguese, aligning with findings that multilingual models benefit low-resource sequence labeling tasks.

Limitations and Future Work

  • Data Imbalance: The paper acknowledges the significant data imbalance across languages, particularly the lack of annotated data for Hindi, which necessitated the use of machine translation.
  • Machine Translation Quality: While machine translation was effective for Hindi, the quality of translations can impact performance, especially for nuanced linguistic phenomena.
  • Zero-Shot Performance: While some zero-shot capabilities were observed, further research could explore methods to systematically improve zero-shot performance for all languages and subtasks.
  • Ensemble Complexity: The paper mentions ensemble systems, but the complexity and computational cost of training and deploying multiple models could be a consideration for real-world applications.

The authors' success in this shared task demonstrates the power of large multilingual transformer models for complex NLP challenges and lays a strong foundation for future work in cross-lingual event detection.

Summary updated

BibTeX
@inproceedings{awasthy2021ibm,
  author = {Parul Awasthy and Jian Ni and Ken Barker and Radu Florian},
  title = {{IBM MNLP IE at CASE 2021 Task 1: Multigranular and Multilingual Event Detection on Protest News}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure