ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence

Yibo Hu, Seyyed MohammadSaleh Hosseini, Erick Skorupa Parolin, Javier Osorio, Latifur Khan, Patrick T. Brandt, Vito D'OrazioNAACL 2022 · DOI

Abstract

Monitoring and analyzing global political violence is a critical but resource-intensive task for political scientists. While general-purpose language models like BERT have advanced automated text analysis, they often struggle with the nuanced language and specialized terminology of conflict research. This paper introduces ConfliBERT, a domain-specific model pre-trained on a massive 33.7 GB corpus of conflict-related text. By employing a custom vocabulary and specialized pre-training strategies, ConfliBERT significantly outperforms general BERT models across 18 distinct tasks, showing particular strength in low-resource environments where labeled data is limited.

Introduction

The study of political violence traditionally relies on manual coding of news reports, a process that is slow, expensive, and difficult to scale. Automated systems offer a solution, but general-purpose models like BERT suffer from a "domain gap." BERT’s training on Wikipedia and books does not prepare it for the strategic nuances of conflict language, where terms like "rebel" or "separatist" carry heavy contextual weight. Furthermore, BERT’s general vocabulary often fragments specialized terms into nonsensical sub-tokens, hindering semantic learning.

Methodology

ConfliBERT was developed using two primary strategies: Pre-training from Scratch (SCR) with a custom "ConfliVocab" of 30,000 tokens, and Continual Pre-training (Cont) of a base BERT model on domain-specific text. The model utilizes the BERT-Base architecture but omits the Next Sentence Prediction (NSP) task, focusing exclusively on Masked Language Modeling (MLM) to better optimize for the domain.

Comparison of ConfliBERT with other domain-specific BERT models and their respective training corpora.

Data and Evaluation

The researchers curated a 33.7 GB training corpus spanning expert domain records (UN, US State Dept), mainstream media (20 GB of news), international newswires, and filtered Wikipedia articles. Evaluation was conducted across 18 tasks on 12 datasets, including binary and multi-class classification and Named Entity Recognition (NER).

Key Results and Insights

ConfliBERT consistently outperformed BERT across nearly all tasks, achieving a lower perplexity (3.14 vs. 3.99) on domain-specific data. A key finding was the importance of the custom vocabulary in preventing word fragmentation, allowing the model to capture semantic meaning more effectively.

Low-Resource Efficiency

Perhaps the most significant advantage of ConfliBERT is its performance in data-scarce environments. In experiments varying the size of the training set, ConfliBERT maintained high F1 scores even with only 1/32 of the available data, whereas general BERT performance dropped sharply. This makes ConfliBERT an essential tool for researchers with limited annotation budgets.

Performance of ConfliBERT variants vs. BERT across different training data scales on GTD, SATP-relevant, and Insight Crime datasets.

Conclusion

ConfliBERT represents a major step forward in the automated analysis of political conflict. By bridging the domain gap through specialized pre-training and vocabulary, it provides a more accurate and efficient tool for political science research. Future work will focus on expanding the model to multilingual contexts and enhancing its ability to handle complex reasoning tasks and zero-shot learning.

Summary updated

BibTeX
@inproceedings{hu2022conflibert,
  author = {Yibo Hu and Seyyed MohammadSaleh Hosseini and Erick Skorupa Parolin and Javier Osorio and Latifur Khan and Patrick T. Brandt and Vito D'Orazio},
  title = {{ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.400},
}
Figure