Abstract
An ever-increasing amount of text, in the form of social media posts and news articles, gives rise to new challenges and opportunities for the automatic extraction of socio-political events. In this paper, we present our submission to the Shared Tasks on Socio-Political and Crisis Events Detection, Task 1, Multilingual Protest News Detection, Subtask 2, Event Sentence Classification, of CASE @ ACL-IJCNLP 2021. In our submission, we utilize the RoBERTa model with additional pretraining, and achieve the best F1 score of 0.8532 in event sentence classification in English and the second-best F1 score of 0.8700 in Portuguese via simple translation. We analyze the failure cases of our model. We also conduct an ablation study to show the effect of choosing the right pretrained language model, adding additional training data and data augmentation.
Introduction
In the era of "big data," social scientists and policymakers are overwhelmed by the sheer volume of online news. Automatically extracting information about socio-political events—like protests—is crucial for timely and proportional responses to changing social circumstances.
However, identifying a "protest event" at the sentence level is challenging. A news article might mention a protest in passing or discuss a "call for a protest" that hasn't happened yet. The core challenge lies in building a classifier that can distinguish between a sentence describing an actual protest event and one that simply uses related vocabulary in a different context.
Methodology
The "NoConflict" team adopted a "backbone-plus-adaptation" approach centered on domain-specific language modeling.
Model Architecture
The system utilizes RoBERTa-base as its backbone. RoBERTa was selected after an ablation study showed it consistently outperformed other architectures like BERT, XLNet, and ELECTRA on this specific task.
Second Pretraining (Domain Adaptation)
The core strategy involved an intermediate round of Masked Language Modeling (MLM) before task-specific fine-tuning. The model was pretrained on the POLUSA dataset, which contains approximately 0.9 million political news articles from 18 outlets across the political spectrum. This allowed the model to "soak up" the nuances, vocabulary, and stylistic conventions of political journalism before seeing the specific training labels.
Fine-tuning and Multilingual Strategy
Following domain adaptation, a simple fully-connected layer was added to the [CLS] token for binary classification. For the Portuguese task, the team employed a "translate-to-English" pipeline using Argos Translate, leveraging the strength of the English-trained model rather than training a native-language model from scratch.
Results and Analysis
The proposed approach achieved the best F1 score of 0.8532 in the English task and the second-best F1 score of 0.8700 in the Portuguese task.
Impact of Domain Adaptation
Ablation studies confirmed that the second pretraining phase on the POLUSA corpus provided a significant performance boost. As shown in the results table, combining all available training data with second pretraining yielded the highest validation F1 score, jumping from 0.8283 to 0.8395.
Key Insights
- Named Entity Signal: Named Entities (NEs) such as city names or political groups are vital features. Stripping NEs from the text led to a significant drop in performance, suggesting the model associates specific entities with protest likelihood.
- Augmentation Trade-offs: While "checklist augmentation" (swapping names/locations) provided minor gains, aggressive methods like paraphrasing were detrimental, often introducing hallucinations or altering sentence meaning.
Limitations and Error Analysis
Despite its performance, the model exhibits three primary failure modes: semantic errors in complex sentences, difficulty distinguishing "calls for protests" (which are excluded by guidelines) from actual events, and contextual dependency where a sentence is ambiguous without its surrounding paragraph.
Conclusion
The "NoConflict" team's success at CASE 2021 demonstrates that domain-specific second pretraining is a powerful tool for socio-political event extraction. Future work will explore incorporating annotation rules directly into model logic and utilizing topological graph structures to provide richer semantic context beyond individual sentences.
Summary updated