Real-Time Discovery and Geospatial Visualization of Mobility and Industry Events from Large-Scale, Heterogeneous Data Streams

Leonhard Hennig, Philippe Thomas, Renlong Ai, Johannes Kirschnick, He Wang, Jakob Pannier, Nora Zimmermann, Sven Schmeier, Feiyu Xu, Jan Ostwald, Hans Uszkoreit2016 · DOI

Abstract

Leonhard Hennig, Philippe Thomas, Renlong Ai, Johannes Kirschnick, He Wang, Jakob Pannier, Nora Zimmermann, Sven Schmeier, Feiyu Xu, Jan Ostwald, Hans Uszkoreit. Proceedings of ACL-2016 System Demonstrations.

2016.

Introduction

In the modern landscape of supply chain management and personal travel planning, the ability to detect disruptive events—such as strikes, layoffs, or traffic accidents—in real-time is paramount. However, the data describing these events is often buried within a massive "firehose" of heterogeneous sources, ranging from informal social media posts to structured RSS feeds and formal news articles.

This paper introduces Spree, a scalable system designed to extract and visualize mobility and industry-related events from these diverse data streams. The authors address three primary challenges: the high volume and velocity of incoming data, the linguistic complexity of German text (particularly in informal contexts), and the extreme ambiguity of domain-specific entities (e.g., distinguishing between identical street names in different cities).

System Architecture

Spree utilizes a decoupled architecture to separate data ingestion from intensive processing tasks. Data is initially gathered via crawlers and pushed to a Kafka message queue. The core processing is handled by Apache Flink, a distributed streaming framework that provides high throughput and robust fault tolerance, effectively managing "back-pressure" when data arrival rates spike.

The Spree system architecture, illustrating the flow from data ingestion to visualization.

Linguistic Pipeline and Event Extraction

The system employs a multi-stage pipeline specifically tailored for German text:

  1. Preprocessing: Tokenization and dependency parsing are performed using the Mate Tools suite.
  2. Named Entity Recognition (NER): The system uses SProUT, combining rule-sets with massive gazetteers to identify companies, streets, and transit routes.
  3. Smart Disambiguation: To resolve location ambiguity, Spree uses a hierarchy-based algorithm. It leverages explicit mentions of cities or "unambiguous" nearby entities to link generic street names to specific geographic coordinates.
  4. Hybrid Event Extraction: For formal news text, the system matches dependency parse trees against learned patterns. For informal Twitter data, it utilizes a high-precision key-phrase-based strategy to overcome the "telegraphic" and noisy nature of social media.

Data Sources and Knowledge Bases

As of early 2016, the system processed approximately 50,000 topically focused tweets, 13,200 news documents, and 136,000 RSS feed items daily. To support precise entity linking, Spree integrates several large-scale knowledge bases, including data on 112,000 German companies, OpenStreetMap data for 27,000 cities and 104,000 streets, and Deutsche Bahn station and route data in GTFS format.

Visualization and Insights

Extracted events are indexed in Solr and presented via an interactive geospatial dashboard. A key innovation of the system is its ability to simulate supply chain impacts. By selecting companies near a detected event and computing routes between them, Spree helps users visualize potential downstream disruptions even when proprietary supplier data is unavailable.

The Spree user interface showing (a) event visualization and (b) a simulated supply chain route affected by local events.

Evaluation and Results

The system was evaluated based on precision across different event types, achieving an average precision of 64%.

  • Source Reliability: RSS feeds proved most reliable, reaching a precision of 1.0 for traffic events due to their formalized language.
  • The Twitter Challenge: While Twitter provided high volume, it suffered from lower precision in certain categories. For example, the term "Stau" (traffic jam) often appeared metaphorically in tweets, leading to a precision of only 0.28 for that specific category on the platform.
  • Industry Events: Events such as "Layoffs" and "Strikes" were detected with high precision (0.66 to 0.76) across both news and social media sources.

Conclusion

Spree demonstrates the feasibility of real-time event discovery from massive, heterogeneous streams by combining robust streaming architectures with hybrid linguistic analysis. While the current implementation is tailored for German and relies heavily on gazetteer coverage, future work aims to extend the pipeline to additional languages and a broader variety of event types to support global supply chain monitoring.

Summary updated

BibTeX
@misc{hennig2016real,
  author = {Leonhard Hennig and Philippe Thomas and Renlong Ai and Johannes Kirschnick and He Wang and Jakob Pannier and Nora Zimmermann and Sven Schmeier and Feiyu Xu and Jan Ostwald and Hans Uszkoreit},
  title = {{Real-Time Discovery and Geospatial Visualization of Mobility and Industry Events from Large-Scale, Heterogeneous Data Streams}},
  year = {2016},
  doi = {10.18653/v1/p16-4007},
}
Figure