Newspaper Signaling for Crisis Prediction

Prajvi Saxena, Sabine Janzen, Wolfgang MaassNAACL 2024 · DOI

TL;DR

This paper introduces MENDEL, a novel multi-lingual and open-domain system for predicting crises by analyzing unstructured newspaper articles in real-time. By leveraging advanced transformer models and a unique two-stage filtering process, MENDEL aims to overcome the limitations of existing crisis prediction systems, offering a more comprehensive and adaptable approach.

What Problem Does This Paper Address?

The timely detection of emerging crises is crucial for organizations and governments to prepare and respond effectively. While newspaper articles are a rich source of information, effectively mining them for crisis signals is challenging due to the sheer volume of unstructured data, the presence of media and cultural biases, and the need to process multiple languages. Existing systems often rely on structured data, limited language support, or specific domains, failing to capture the nuances of global crisis signaling. This paper addresses the gap by proposing a system that can handle diverse, unstructured news data across languages and domains to identify early crisis indicators.

Methods

MENDEL employs a multi-stage pipeline built upon state-of-the-art transformer models:

  • Data Acquisition: The system starts by taking user-defined keywords and parameters (language, country, time frame) to query news APIs. It utilizes OpenAI's GPT-3.5 for keyword expansion to broaden the search scope.
  • Data Processing: Raw news articles undergo cleaning, including punctuation and stop-word removal. STANZA is then used for tokenization and lemmatization, breaking down text into its core components.
  • Two-Stage Data Filtration: This is a key innovation.
  • The first stage uses a multilingual RoBERTa model to filter articles based on domain relevance, employing cosine similarity between article and keyword embeddings.
  • The second stage uses a zero-shot multilingual XLM-RoBERTa model to specifically identify articles discussing future or present warnings, filtering out past-tense events.
  • Context-based Reasoning and Forecasting: The filtered articles are then classified into 'risk and warning', 'caution and advice', or 'safe and harmless' categories using GPT-3.5. The system calculates the percentage of 'risk and warnings' to generate alerts and visualizes trends over a six-month period. KeyBERT is used to extract relevant keywords from high-risk articles, and an extractive QA chatbot is provided for user interaction.

Datasets

The paper describes an experimental evaluation using a dataset of 18,673 news articles collected in real-time from July 2021 to May 2023 in Germany. These articles were related to three specific crisis events: economic recession, high energy prices, and high gasoline prices. For evaluation, a subset of 319 articles from the filtered data was used to create a ground truth corpus by recruiting three human annotators.

Key Insights and Ideas

  • Leveraging Unstructured Data: The core idea is that purely unstructured newspaper data, when processed effectively, can reveal early crisis signals without relying on pre-structured, potentially biased, knowledge bases.
  • Multi-lingual and Open-Domain Capability: MENDEL's ability to handle over 66 languages and adapt to various crisis domains (energy, finance, supply chains) is a significant step towards a universal crisis signaling tool.
  • The Power of Two-Stage Filtration: The paper highlights the critical importance of a multi-layered filtering approach. Simply aggregating news is insufficient; filtering for domain relevance and future/present warnings is essential to extract actionable insights from noisy data.
  • Human-in-the-Loop for Evaluation: The use of human annotators to create a ground truth dataset for evaluation underscores the complexity of crisis signaling and the need for robust validation, especially in the absence of large, pre-existing crisis benchmarks.

Key Results

  • MENDEL was able to detect early signals for economic recession and energy-related crises at intervals of 4, 8, and 12 weeks before the events occurred.
  • The system achieved an average micro-F1 score of up to 80% in classifying articles into 'risk and warning', 'caution and advice', or 'safe and harmless' categories, as evaluated against human-annotated data.
  • The two-stage data filtration process was crucial, with an average of 67.37% of collected articles deemed irrelevant to the selected domains and future signals.
  • For the 'high energy prices' event, the system showed a growing trend in risk and warning signals, indicating its predictive capability.

Limitations and Future Work

  • Verification of Warnings: MENDEL is designed to offer early warning signs, not to definitively verify crises. The authors acknowledge that verification is a task for future work.
  • Performance Variability: The paper notes that the system's performance is highly influenced by the quality and quantity of news articles available, and not all crises may be predictable with the same level of performance.
  • Future Work: The authors suggest further research into improving the verification of crisis warnings and exploring how to enhance the system's ability to deal with unstructured data for more robust crisis prediction.

Summary updated

BibTeX
@inproceedings{saxena2024newspaper,
  author = {Prajvi Saxena and Sabine Janzen and Wolfgang Maass},
  title = {{Newspaper Signaling for Crisis Prediction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-demo.17},
}
Figure