Abstract
Salvatore Giorgi, Vanni Zavarella, Hristo Tanev, Nicolas Stefanovitch, Sy Hwang, Hansi Hettiarachchi, Tharindu Ranasinghe, Vivek Kalyan, Paul Tan, Shaun Tan, Martin Andrews, Tiancheng Hu, Niklas Stoehr, Francesco Ignazio Re, Daniel Vegh, Dennis Atzenhofer, Brenda Curtis, Ali Hürriyetoğlu. Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021).
2021.
Introduction: The Utility Gap in Event Extraction
Automated event extraction (EE) systems are traditionally evaluated on their ability to identify events within isolated sentences or documents. While these benchmarks measure linguistic precision, they often fail to capture a system's real-world utility for social scientists or policy-makers. This study, conducted as part of Shared Task 3 at CASE 2021, shifts the focus toward spatio-temporal modeling. The core question is whether automated systems can accurately reproduce a map of the 2020 Black Lives Matter (BLM) protests that matches ground truth data.
Methods and Systems
Participants were tasked with extracting protest incidents (date and location) from raw text. Several distinct approaches were evaluated:
- Baseline (NEXUS): A rule-based engine utilizing cascaded grammars and geocoding via the GeoNames gazetteer.
- NexusDdpl: An extension of the baseline incorporating a deduplication module. It leveraged LASER embeddings for semantic similarity and geographical distance to merge redundant reports.
- Transformer-based Models: Teams utilized architectures like BERTweet, RoBERTa, and XLM-RoBERTa, fine-tuned on socio-political event data.
- Hybrid Filtering (NoConflict): One of the top-performing systems combined RoBERTa classification with curated keyword filters and section-name constraints (e.g., focusing on the "U.S." or "Politics" sections of the NYT).
Datasets: The Gold Standard vs. The Noise
The study utilized three primary data sources:
- News Input: 5,347 articles from The New York Times (May 25 – June 30, 2020).
- Social Media Input: A subset of ~32,000 geotagged tweets from a larger collection of 30 million BLM-related posts.
- The Gold Standard: A manually curated dataset of 7,976 protest events compiled from sources including Creosote Maps and the Crowd Counting Consortium (CCC).
Key Results: Temporal Success vs. Spatial Struggle
The evaluation revealed a significant disparity between tracking a movement's "pulse" and mapping its specific geography. Systems were evaluated using correlation coefficients (Pearson $r$ and Spearman $\rho$) against the Gold Standard.
Daily Trends
Automated systems proved highly reliable for monitoring the frequency of protests over time. The best system (NoConflict) achieved a Spearman $\rho$ of 0.762 for tracking daily protest counts. This suggests that EE systems can effectively capture the temporal evolution of a social movement.
Spatio-Temporal Mapping and the Recall Gap
Performance plummeted when systems were required to map events to specific 55km grid squares. The highest Pearson $r$ for "cell-day" counts was only 0.210. This difficulty is compounded by a massive recall gap; even the best system identified only 5.08% of the actual protests recorded in the Gold Standard.
Discussion and Future Directions
The primary bottleneck identified was media sourcing. National outlets like The New York Times and sparse geotagged tweets are insufficient for capturing the thousands of localized protests that define a mass movement. Many events in smaller towns left no national media footprint, leading to the observed low recall.
To improve discovery in future social movements, the authors suggest:
- Integrating local news sources to capture events outside major metropolitan areas.
- Enhancing geocoding for social media, where precise GPS data is often unavailable.
- Developing cross-modal reconciliation to link disparate reports (e.g., a tweet and a news article) to the same physical event.
Summary updated