SU-NLP at CASE 2021 Task 1: Protest News Detection for English

Furkan Çelik, Tuğberk Dalkılıç, Fatih Beyhan, Reyyan YeniterziACL 2021

TL;DR

This paper details Sabancı University's participation in the CASE 2021 shared task on protest news detection. They tackled four subtasks in English, achieving strong results by combining transformer models with novel architectures like CNN-LexStem and transformer-LSTM-CRF, and employing ensemble methods for improved robustness.

What Problem Does This Paper Address?

The paper addresses the challenge of automatically identifying and extracting information about socio-political events, specifically protest events, from news articles. This involves several sub-tasks: determining if a document or sentence contains an event, resolving event coreferences (identifying when different sentences refer to the same event), and extracting event triggers and their arguments. Accurately performing these tasks is crucial for understanding public discourse and societal trends, but it's complicated by the nuances of natural language, including varied sentence structures, tenses, and the potential for ambiguous or indirect event mentions.

Methods

The researchers explored several approaches across the four subtasks:

  • Event Detection (Subtasks 1 & 2): They used standard transformer models (BERT, ALBERT, RoBERTa) and explored ensemble methods. A novel approach involved a multi-channel CNN architecture called LexStem. This model uses two channels: one for the original word forms and another for their stemmed versions, aiming to capture variations in tense and morphology. They also experimented with CNN-StemStem to isolate the effect of stemming.
  • Event Coreference Resolution (Subtask 3): A two-step approach was employed: first, a binary classifier (using BERT, ALBERT, RoBERTa, and an ensemble) to identify pairs of sentences referring to the same event, and second, a clustering algorithm (based on Örs et al., 2020) to group coreferent sentences. They found that using an extended dataset (combining CASE and AESPEN data) significantly improved classification performance.
  • Event Extraction (Subtask 4): They baseline with BERT and RoBERTa, and then proposed a transformer-BiLSTM-CRF architecture. This combines the power of transformers for contextual understanding with BiLSTM for sequence modeling and CRF for sequence labeling, a combination that has shown success in Named Entity Recognition.

Datasets

  • CASE 2021 Shared Task Data: Provided datasets for English, Portuguese, Spanish, and Hindi. The paper focuses solely on the English portion.
  • AESPEN'20 Shared Task Data: An external dataset for event sentence coreference identification was used to augment the CASE data for Subtask 3, creating an "EXT" version of the dataset.

Key Insights and Ideas

  • Stemming Helps: The LexStem model, which incorporates stemmed word forms alongside original forms in a CNN architecture, showed significant improvements, particularly for sentence-level event detection (Subtask 2). This suggests that morphological variations are important cues for identifying events.
  • Transformers Reign Supreme (Mostly): Transformer-based models (BERT, RoBERTa) consistently outperformed traditional neural networks (CNN, LSTM, GRU) across most tasks, highlighting their effectiveness in capturing complex linguistic patterns.
  • Ensembles for Robustness: Ensemble models, particularly for Subtask 2 (event detection) and Subtask 3 (coreference classification), provided performance gains and increased robustness, especially when the primary model (RoBERTa) was less confident in its predictions.
  • Transformer + BiLSTM + CRF is Powerful: The transformer-BiLSTM-CRF architecture proved highly effective for event extraction (Subtask 4), outperforming simpler transformer baselines. This hybrid approach leverages the strengths of each component.
  • Data Augmentation Matters: For coreference resolution, using an extended dataset (CASE + AESPEN) led to substantial improvements in classification accuracy, underscoring the value of larger, diverse training data.

Key Results

  • Subtask 1 & 2 (Event Detection): The proposed CNN-LexStem model achieved competitive results, with the ensemble of RoBERTa and other models showing strong performance.
  • Subtask 3 (Event Coreference): The ensemble model, trained on the extended dataset, achieved the highest F1 scores, reaching 80.54% on the validation set for classification and 78.66% CONLL score for clustering.
  • Subtask 4 (Event Extraction): The RoBERTa-BiLSTM-CRF model achieved the highest F1 scores, reaching 0.76 on both validation and test sets. The authors believe this model would have ranked second if not for a minor submission file format issue.

Limitations and Future Work

  • Language Focus: The study was limited to English due to time constraints. Future work will explore other languages to see if the observed trends hold.
  • Ensemble Complexity: While ensembles improved performance, they increase model complexity. Further research could explore more efficient ensemble strategies.
  • Coreference Classification Boundary: For Subtask 3, a threshold of 0.6 was used instead of the standard 0.5 for positive labels, which yielded better results. Understanding the implications of this boundary choice across different datasets is a potential area for future exploration.
  • Event Extraction Nuances: In Subtask 4, the model performed better at identifying event triggers than arguments. The confusion matrix revealed that "event time" was well-identified, while "organizer" and "participant" tags were frequently confused, suggesting areas for targeted improvement.

Summary updated

BibTeX
@inproceedings{celik2021su,
  author = {Furkan Çelik and Tuğberk Dalkılıç and Fatih Beyhan and Reyyan Yeniterzi},
  title = {{SU-NLP at CASE 2021 Task 1: Protest News Detection for English}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure