Geolocating Political Events in Text

Andrew HaltermanNAACL 2019

TL;DR

This paper introduces a novel method for automatically identifying the specific locations where political events described in text occurred. By training a neural network on a new dataset of 8,000 labeled sentences, the authors achieve human-level performance, outperforming previous systems and enabling more granular analysis of global events for social scientists.

What Problem Does This Paper Address?

Social scientists, particularly those in comparative politics and security studies, are increasingly interested in subnational-level data to understand complex phenomena. Simultaneously, the vast amount of text data available online presents a rich source of information. However, a significant challenge lies in connecting these two trends: automatically linking specific events mentioned in text to their precise geographical locations. Existing methods often either return a single, generalized location for an entire document or struggle with sentences containing multiple events and locations. This paper aims to bridge this gap by developing a system that can accurately pinpoint the location of each event within a sentence.

Methods

The core of this work is a recurrent neural network (RNN) designed for the "event-location linking" task. Given a sentence and a specific event (identified by its verb), the model predicts which words in the sentence represent the location where that event took place.

Key methodological aspects include:

  • Task Formalization: The task is defined as assigning a binary label to each word in a sentence, indicating whether it's a location for a specific event.
  • Neural Network Architecture: The authors explore two neural network models: a Convolutional Neural Network (CNN) with residual layers and a Bidirectional Long Short-Term Memory (LSTM) network. The LSTM model ultimately performs better.
  • Feature Engineering: The models utilize a rich set of features derived from the spaCy NLP library, including GloVe word embeddings, dependency labels, named entity tags, part-of-speech tags, and crucially, the distance (both linear and dependency-based) between a word and the event's verb.
  • Baseline Models: The authors compare their neural network models against a simple rule-based baseline (closest location to the event verb) and existing approaches like Profile (Imani et al., 2017) and PropBank's ArgM-LOC label.

Datasets

  • Newly Introduced Dataset: The paper introduces a new dataset of approximately 8,000 hand-labeled English sentences. Each sentence is annotated with an event verb and its corresponding location(s).
  • Data Sources: The sentences are drawn from a diverse range of sources, including international news wires (50%), local Syrian media (35%), and non-news sources like Wikipedia and atrocity monitoring reports (15%).
  • Annotation Granularity: Annotations capture the most specific named places, allowing for multiple locations per event and zero locations for events not tied to a specific place.
  • Evaluation Datasets: The models are evaluated on the newly created dataset, as well as implicitly compared to results from previous work on datasets like PropBank and ICEWS.

Key Insights and Ideas

  • Event-Specific Geolocation is Crucial: Simply finding the "main" location of a sentence is insufficient. Accurately linking each event to its specific location is vital for detailed analysis, especially in complex texts with multiple events.
  • Rich Linguistic Features Matter: While word embeddings are useful, incorporating features like dependency labels and part-of-speech tags significantly improves performance. This highlights the importance of grammatical structure and semantic roles in understanding location context.
  • LSTM Outperforms CNN for This Task: The LSTM model, with its ability to model sequential dependencies, proved more effective than the CNN for capturing the nuances of event-location relationships in text.
  • Human-Level Performance Achieved: The model not only surpasses previous automated methods but also matches or exceeds human annotator performance, suggesting its readiness for practical application.
  • Broader Applicability: The method is designed to be general and can be applied to various event extraction systems and geographic contexts, moving beyond domain-specific solutions.

Key Results

  • Per-Token Performance: The best-performing LSTM model achieved a per-token precision, recall, and F1 score of 0.85, 0.83, and 0.84, respectively.
  • Full-Sentence Accuracy: The LSTM model achieved a full-sentence accuracy of 0.77.
  • Comparison to Baselines: The LSTM significantly outperformed the word distance baseline (F1: 0.27), Profile (F1: 0.37), and PropBank (F1: 0.47).
  • Comparison to Prior Work: The LSTM's performance is comparable to or better than the reported results for Chung et al. (2017) (F1: 0.62), and significantly exceeds the human annotator's F1 score of 0.74.
  • Ablation Study: The ablation test revealed that while GloVe embeddings are helpful, they are not sufficient on their own. Part-of-speech features were found to be more useful than dependency labels, and named entity recognition features did not improve accuracy, possibly due to dataset skew.

Limitations and Future Work

  • Limitations:
    • The model sometimes fails to select the most specific location when multiple are available.
  • The ablation study showed variability in feature importance across different data splits, suggesting caution in drawing definitive conclusions about individual feature contributions.
  • The current model is sentence-based; extending it to handle longer contexts could improve accuracy.
  • Future Work:
    • Integrate Toponym Resolution: Combine the event-location linking step with toponym resolution (identifying and disambiguating place names) for improved accuracy.
  • Extend to Longer Contexts: Develop models that can process text beyond a single sentence to capture event-location relationships across paragraphs or documents.
  • Replace Categorical Features with Embeddings: Explore using embeddings for categorical features like POS and dependency labels to potentially capture richer semantic information.
  • Broader Event Types: Extend the model to handle a wider range of event types beyond those focused on territorial capture.

Summary updated

BibTeX
@inproceedings{halterman2019geolocating,
  author = {Andrew Halterman},
  title = {{Geolocating Political Events in Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure