SAGE: A Hybrid Geopolitical Event Forecasting System

Fred Morstatter, Aram Galstyan, Gleb Satyukov, Daniel Benjamin, Andrés Abeliuk, Mehrnoosh Mirtaheri, K. S. M. Tozammel Hossain, Pedro A. Szekely, Emilio Ferrara, Akira Matsui, Mark Steyvers, Stephen Bennett, David V. Budescu, Mark Himmelstein, Michael D. Ward, Andreas Beger, Michele Catasta, Rok Sosic, Jure Leskovec, Pavel Atanasov, Regina Joseph, Rajiv Sethi, Ali E. Abbas

Introduction

Predicting geopolitical events—ranging from market fluctuations to election outcomes and military conflicts—remains a formidable challenge. Traditional subject-matter experts often struggle to outperform random baselines, while "wisdom of the crowds" approaches, though effective, frequently fail to account for historical "base rates" or process vast quantitative datasets efficiently. Conversely, machine learning models excel at data aggregation but lack the qualitative reasoning and contextual understanding inherent to human experts.

SAGE (Synergistic Anticipation of Geopolitical Events) addresses this gap by creating a hybrid forecasting platform. By combining human intuition with machine-generated statistical anchors, SAGE aims to produce more accurate forecasts than either humans or machines could achieve in isolation.

Hybridization Strategies

SAGE facilitates collaboration through two primary strategies tailored to the nature of the forecasting question:

Quantitative: Time Series Forecasting

For questions involving numerical outcomes, such as financial indices or commodity prices, SAGE provides historical data sourced from Wikipedia and Trading Economics. The system applies models like ARIMA, Random Walk, and Recurrent Neural Networks (RNNs) to project values forward. A key feature of the interface is its interactivity: forecasters can adjust model parameters and training periods to observe how different algorithmic assumptions influence the prediction.

Qualitative: CAMEO Base Rate Visualization

For events lacking a direct numerical history (e.g., the likelihood of a specific military attack), SAGE utilizes the CAMEO (Conflict and Mediation Event Observations) framework. By querying historical event data from the Integrated Crisis Early Warning System (ICEWS), the platform generates heatmaps showing the frequency of similar past events. This statistical "anchor" helps human forecasters overcome cognitive biases, such as the tendency to overreact to recent sensational news while ignoring long-term historical frequencies.

Data Integration and Evaluation

The platform integrates several external data streams to support its hybrid approach:

  • Financial/Commodity Data: Sourced from Trading Economics and Wikipedia for time-series modeling.
  • Geopolitical Event Data: Utilizing the ICEWS repository for CAMEO-based visualizations.
  • Crowdsourced Forecasts: SAGE has engaged over 500 human forecasters who have addressed more than 450 questions across geopolitics, medicine, and climate.

Forecasts are submitted as probability distributions and evaluated using the Brier score. This metric penalizes overconfidence in incorrect outcomes, providing a rigorous framework for tracking accuracy.

Key Insights and Deployment

The SAGE platform emphasizes machine learning as a reasoning tool rather than a simple oracle. By visualizing base rates and allowing users to toggle between different models (e.g., ARIMA vs. RNN), the system encourages "active" forecasting. This interaction allows humans to critique the machine's logic and integrate qualitative nuances that algorithms might miss.

While this system demonstration focuses on architecture, the deployment has proven the platform's flexibility across diverse domains, including sports and medicine. The use of the Brier score ensures that the platform maintains a mathematical standard for improvement.

Limitations and Future Directions

Despite its strengths, SAGE faces challenges in scenarios with extreme data scarcity where historical precedents are unavailable. Additionally, optimizing the human-computer interaction (HCI) to maximize accuracy remains an ongoing research focus. Future iterations may incorporate sophisticated Natural Language Processing (NLP) models to automatically parse news feeds and directly inform the human reasoning workflow.

Summary updated

Abstract

Forecasting of geopolitical events is a notoriously difficult task, with experts failing to significantly outperform a random baseline across many types of forecasting events. One successful way to increase the performance of forecasting tasks is to turn to crowdsourcing: leveraging many forecasts from non-expert users. Simultaneously, advances in machine learning have led to models that can produce reasonable, although not perfect, forecasts for many tasks. Recent efforts have shown that forecasts can be further improved by ``hybridizing'' human forecasters: pairing them with the machine models in an effort to combine the unique advantages of both. In this demonstration, we present Synergistic Anticipation of Geopolitical Events (SAGE), a platform for human/computer interaction that facilitates human reasoning with machine models.
BibTeX
@inproceedings{morstatter2019sage,
  author = {Fred Morstatter and Aram Galstyan and Gleb Satyukov and Daniel Benjamin and Andrés Abeliuk and Mehrnoosh Mirtaheri and K. S. M. Tozammel Hossain and Pedro A. Szekely and Emilio Ferrara and Akira Matsui and Mark Steyvers and Stephen Bennett and David V. Budescu and Mark Himmelstein and Michael D. Ward and Andreas Beger and Michele Catasta and Rok Sosic and Jure Leskovec and Pavel Atanasov and Regina Joseph and Rajiv Sethi and Ali E. Abbas},
  title = {{SAGE: A Hybrid Geopolitical Event Forecasting System}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/955},
}
Figure