ALEM at CASE 2021 Task 1: Multilingual Text Classification on News Articles

Alaeddin Gürel, Emre EminACL 2021

TL;DR

This paper presents ALEM's approach to the CASE 2021 Task 1, focusing on multilingual text classification of sociopolitical and crisis events in news articles. The team explored fine-tuning pre-trained BERT models for English, Spanish, and Portuguese, investigating various data preprocessing and modeling strategies to tackle challenges like imbalanced datasets and BERT's token length limitations.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically identifying and classifying sociopolitical and crisis event information from news articles across multiple languages. This is crucial for understanding global events, tracking crises, and informing policy. Specifically, the CASE 2021 Task 1 aimed to:

  • Document Classification: Determine if a news article contains at least one past or ongoing event.
  • Sentence Classification: Identify sentences that contain event triggers.
  • Event Sentence Coreference Identification: Link sentences that refer to the same event.
  • Event Extraction: Extract specific details about events (e.g., time, participants, location).

The authors highlight the inherent difficulties in this task, including the need for multilingual capabilities, handling imbalanced data distributions across languages and event categories, and overcoming technical constraints like BERT's maximum input token length.

Methods

The ALEM team primarily employed fine-tuning pre-trained BERT models for their approach. Key methodological aspects include:

  • Multilingual BERT (mBERT): For multilingual tasks, they utilized the pre-trained mBERT model, concatenating data from English, Spanish, and Portuguese to train a single model.
  • Language-Specific BERT Models: For monolingual tasks, they fine-tuned BERT models pre-trained on specific languages:
  • English: BERT Base
    • Spanish: BETO
    • Portuguese: BERTimbau Base
  • Data Preprocessing:
    • Imbalanced Data Handling: They experimented with oversampling and undersampling techniques using the imbalanced-learn package. However, these methods showed negligible impact, leading them to opt for naive random sampling.
  • Token Length Limitation: To address BERT's 512-token limit, they explored a "populating method" where long documents were split into chunks, and each chunk was assigned the parent document's label. This was done for both training and prediction, with predictions aggregated by majority vote. While this method didn't significantly improve results, it was a strategy to mitigate data loss.
  • Model Training: They fine-tuned models with varying epoch numbers and batch sizes to optimize hyperparameters, settling on specific configurations for BERT Base and mBERT models for document and sentence classification tasks.

Datasets

The shared task provided training data in English, Spanish, and Portuguese for each subtask. The data was structured in JSON Lines format for document and sentence classification, containing IDs, text, and labels. For event extraction, a CoNLL-like format was used for token-level data.

The paper notes that the English dataset was significantly larger than the Spanish and Portuguese datasets for both document and sentence classification. For event extraction, there was insufficient data for Spanish and Portuguese to train robust models.

Key Insights and Ideas

  • Language-Specific Models vs. Multilingual: While mBERT was used for multilingual tasks, the paper found that for sentence classification, language-specific BERT models (BERT Base for English, BETO for Spanish, BERTimbau for Portuguese) generally outperformed mBERT. This suggests that specialized pre-training can be beneficial even when aiming for multilingual performance.
  • Data Imbalance Impact: Despite significant efforts to address data imbalance through over/undersampling, these techniques had minimal impact on the results. This points to the inherent difficulty of imbalance in real-world NLP tasks and suggests that other factors might be more dominant.
  • Token Length Mitigation: The strategy of splitting long documents into smaller chunks for BERT processing, while not improving results, highlights a common practical challenge in applying transformer models to long texts. The observation that this didn't significantly harm performance is also noteworthy.
  • Event Extraction Data Scarcity: The lack of sufficient data for Spanish and Portuguese in event extraction underscores the challenges of building robust NLP systems for low-resource languages and specific, fine-grained tasks.

Key Results

The paper reports F1-macro scores for document and sentence classification, and F1 scores for event extraction (token level).

  • Document Classification (F1-macro):
    • English BERT: 84.26%
    • Spanish BETO: 73.82%
    • Portuguese BERTimbau: 80.20%
    • Multilingual BERT (mBERT): 84.17% (English), 76.32% (Spanish), 79.78% (Portuguese)
    • Insight: For English, BERT Base slightly outperformed mBERT. For Spanish, BETO was significantly better than mBERT. For Portuguese, BERTimbau was slightly better than mBERT.
  • Sentence Classification (F1-macro):
    • English BERT: 87.68%
    • Spanish BETO: 83.95%
    • Portuguese BERTimbau: 82.72%
    • Multilingual BERT (mBERT): 84.70% (English), 76.53% (Spanish), 82.01% (Portuguese)
    • Insight: Language-specific BERT models generally outperformed mBERT for sentence classification across all languages.
  • Event Extraction (F1-score for English):
    • etime: 77.95%
    • fname: 54.65%
    • organizer: 65.89%
    • participant: 75.40%
    • place: 83.86%
    • target: 55.10%
    • trigger: 84.32%
    • Insight: Performance varied significantly across event categories, with "place" and "trigger" showing higher scores, while "fname" and "target" were more challenging.

Limitations and Future Work

  • Time Constraints: The authors explicitly mention that time limitations prevented them from completing all experiments, particularly for subtask 3 (event sentence coreference identification). They also had to limit their submission to only one attempt per subtask.
  • Data Augmentation: They plan to explore data augmentation techniques (e.g., text generation) in future work to address data scarcity and imbalance, though they anticipate this might be more effective for English due to data availability.
  • Ensemble Learning: Ensemble methods are considered for future experiments, aiming to combine predictions from multiple models to improve robustness and accuracy. This could involve building systems on top of BERT embeddings.
  • Token Length Handling: While they implemented a strategy to handle BERT's token limit, they acknowledge that this is a significant limitation and suggest further research into more effective pre-processing or model architectures for long documents.
  • Event Extraction Data: The lack of sufficient data for Spanish and Portuguese in event extraction is a clear limitation, necessitating different approaches for future training and evaluation in these languages.

Summary updated

BibTeX
@inproceedings{gurel2021alem,
  author = {Alaeddin Gürel and Emre Emin},
  title = {{ALEM at CASE 2021 Task 1: Multilingual Text Classification on News Articles}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure