PROTEST-ER: Retraining BERT for Protest Event Extraction

Tommaso Caselli, Osman Mutlu, Angelo Basile, Ali HürriyetoğluACL 2021

TL;DR

This paper introduces PROTEST-ER, a BERT model specifically retrained on protest-related news data to improve event extraction. By using domain adaptive retraining (DAR), PROTEST-ER significantly outperforms a generic BERT model on out-of-domain protest event datasets, demonstrating the effectiveness of targeted pre-training for enhancing model portability.

What Problem Does This Paper Address?

Extracting socio-political events from text is crucial for understanding societal dynamics, but existing Natural Language Processing (NLP) models struggle with portability. This means models trained on one dataset (e.g., news articles from one country) often perform poorly when applied to similar data from a different domain or context (e.g., news from another country or about a different type of event). This "domain adaptation" problem is particularly challenging for event extraction, where subtle differences in language, topics, and cultural contexts can lead to significant performance drops. The authors aim to improve the portability of event extraction models by developing a strategy to adapt pre-trained language models like BERT to the specific domain of protest events.

Methods

The core of this paper is Domain Adaptive Retraining (DAR). This technique involves taking a pre-trained language model (like BERT) and further training it on a large corpus of text that is relevant to the target domain. In this case, the target domain is protest events, and the authors use two sets of news articles: one general news corpus (TREC Washington Post Corpus version 2) and one specifically focused on protest news (TREC Washington Post Corpus version 3, protest-related documents).

The process involves:

  1. Further Pre-training: BERT is retrained using the Masked Language Modeling (MLM) objective on the collected protest-related data. This creates two new models: NEWS-BERT (trained on general news) and PROTEST-ER (trained on protest-specific news).
  2. Fine-tuning: These retrained models, along with the original BERT, are then fine-tuned on the specific task of protest event extraction using labeled data from the CLEF ProtestNews Lab.
  3. Evaluation: The performance of these models is evaluated on both in-domain (India) and out-of-domain (China) test sets to assess their portability.

The authors also explore different input granularities for fine-tuning, comparing document-level and sentence-level processing.

Datasets

  • TREC Washington Post Corpus (version 2 & 3): Used for unsupervised domain adaptive retraining. Version 2 (WPC-Gen) contains general news articles, while version 3 (WPC-Ev) contains protest-related news articles.
  • CLEF 2019 ProtestNews Lab Data: This dataset, split into India (source) and China (target) varieties, is used for fine-tuning and evaluating the event extraction models. It includes document classification, sentence classification, and event extraction tasks.

Key Insights and Ideas

  • Targeted Pre-training Matters: Simply retraining BERT on general news (NEWS-BERT) doesn't yield as significant improvements as retraining on protest-specific news (PROTEST-ER). This highlights that the content of the domain-adaptive data is crucial.
  • Portability is Achievable: PROTEST-ER demonstrates a substantial improvement in performance on out-of-domain data, suggesting that domain adaptation can effectively bridge the gap between different varieties of text within the same genre.
  • Sentence-level Input Can Be Effective: For PROTEST-ER, processing data at the sentence level during fine-tuning proved to be more effective than document-level processing for the India dataset, indicating that finer-grained input can help models leverage their adapted representations.
  • Arguments are Harder than Triggers: The results consistently show that identifying event arguments is more challenging than identifying event triggers, with larger performance drops observed for arguments when moving to out-of-domain data.

Key Results

  • PROTEST-ER outperforms a generic BERT model by 8.1 points on out-of-domain data.
  • The best performing models (PROTEST-ER) achieve 51.91 F1-score on the India dataset and 46.39 F1-score on the China dataset.
  • PROTEST-ER achieves competitive scores with only 10% of the training data, whereas generic BERT requires 30% (document input) or 70% (sentence input) to reach comparable performance.
  • For the India dataset, PROTEST-ER with sentence input achieved the best results. Interestingly, for the China dataset, document input surprisingly yielded the best results for PROTEST-ER.

Limitations and Future Work

  • Performance Gap with State-of-the-Art: Despite improvements, PROTEST-ER still falls short of the best-performing system from the CLEF ProtestNews challenge (a BiLSTM-CRF model). The authors are unsure if this is due to the retraining process or the limited training corpus.
  • Investigating Data Size: The authors plan to further investigate the impact of training data size on TLM performance.
  • Multilingual Extensions: Exploring multilingual versions of PROTEST-ER is a key direction for future work.
  • Document vs. Sentence Input: Further research is needed to understand why document-level input unexpectedly performed best for PROTEST-ER on the China dataset, especially given the success of sentence-level input for India.

Summary updated

BibTeX
@inproceedings{caselli2021protest,
  author = {Tommaso Caselli and Osman Mutlu and Angelo Basile and Ali Hürriyetoğlu},
  title = {{PROTEST-ER: Retraining BERT for Protest Event Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure