Abstract
Leonhard Hennig, Philippe Thomas, Renlong Ai, Johannes Kirschnick, He Wang, Jakob Pannier, Nora Zimmermann, Sven Schmeier, Feiyu Xu, Jan Ostwald, Hans Uszkoreit. Proceedings of ACL-2016 System Demonstrations.
2016.
Introduction
In the modern landscape of supply chain management and personal travel planning, the ability to detect disruptive events—such as strikes, layoffs, or traffic accidents—in real-time is paramount. However, the data describing these events is often buried within a massive "firehose" of heterogeneous sources, ranging from informal social media posts to structured RSS feeds and formal news articles.
This paper introduces Spree, a scalable system designed to extract and visualize mobility and industry-related events from these diverse data streams. The authors address three primary challenges: the high volume and velocity of incoming data, the linguistic complexity of German text (particularly in informal contexts), and the extreme ambiguity of domain-specific entities (e.g., distinguishing between identical street names in different cities).
System Architecture
Spree utilizes a decoupled architecture to separate data ingestion from intensive processing tasks. Data is initially gathered via crawlers and pushed to a Kafka message queue. The core processing is handled by Apache Flink, a distributed streaming framework that provides high throughput and robust fault tolerance, effectively managing "back-pressure" when data arrival rates spike.
Linguistic Pipeline and Event Extraction
The system employs a multi-stage pipeline specifically tailored for German text:
- Preprocessing: Tokenization and dependency parsing are performed using the Mate Tools suite.
- Named Entity Recognition (NER): The system uses SProUT, combining rule-sets with massive gazetteers to identify companies, streets, and transit routes.
- Smart Disambiguation: To resolve location ambiguity, Spree uses a hierarchy-based algorithm. It leverages explicit mentions of cities or "unambiguous" nearby entities to link generic street names to specific geographic coordinates.
- Hybrid Event Extraction: For formal news text, the system matches dependency parse trees against learned patterns. For informal Twitter data, it utilizes a high-precision key-phrase-based strategy to overcome the "telegraphic" and noisy nature of social media.
Data Sources and Knowledge Bases
As of early 2016, the system processed approximately 50,000 topically focused tweets, 13,200 news documents, and 136,000 RSS feed items daily. To support precise entity linking, Spree integrates several large-scale knowledge bases, including data on 112,000 German companies, OpenStreetMap data for 27,000 cities and 104,000 streets, and Deutsche Bahn station and route data in GTFS format.
Visualization and Insights
Extracted events are indexed in Solr and presented via an interactive geospatial dashboard. A key innovation of the system is its ability to simulate supply chain impacts. By selecting companies near a detected event and computing routes between them, Spree helps users visualize potential downstream disruptions even when proprietary supplier data is unavailable.
Evaluation and Results
The system was evaluated based on precision across different event types, achieving an average precision of 64%.
- Source Reliability: RSS feeds proved most reliable, reaching a precision of 1.0 for traffic events due to their formalized language.
- The Twitter Challenge: While Twitter provided high volume, it suffered from lower precision in certain categories. For example, the term "Stau" (traffic jam) often appeared metaphorically in tweets, leading to a precision of only 0.28 for that specific category on the platform.
- Industry Events: Events such as "Layoffs" and "Strikes" were detected with high precision (0.66 to 0.76) across both news and social media sources.
Conclusion
Spree demonstrates the feasibility of real-time event discovery from massive, heterogeneous streams by combining robust streaming architectures with hybrid linguistic analysis. While the current implementation is tailored for German and relies heavily on gazetteer coverage, future work aims to extend the pipeline to additional languages and a broader variety of event types to support global supply chain monitoring.
Summary updated