TL;DR
This paper introduces ApproxSSPS, a novel system designed to efficiently process massive, real-time spatial data streams from smart cities. It tackles the challenge of fluctuating data rates and skewed distributions by employing QoS-aware approximate query processing, striking a balance between latency and accuracy to provide timely insights.
What Problem Does This Paper Address?
Smart cities generate enormous volumes of geo-referenced data from sources like IoT devices, particularly from mobile entities such as vehicles. This data is characterized by high skewness and fluctuating arrival rates, making it difficult for traditional data stream processing systems to keep up while meeting strict latency and accuracy requirements. Existing systems often resort to strategies like backpressure, which can lead to data loss and reduced accuracy, or over-provisioning resources, which is costly. There's a need for systems that can intelligently handle these dynamic data streams, ensuring timely and accurate results for critical smart city applications without compromising stability.
Methods
The core of this work is the ApproxSSPS system, built on Apache Spark Structured Streaming. It introduces a QoS controller that dynamically adjusts the sampling rate of incoming data to meet user-defined latency and/or accuracy targets. The system comprises five key components:
- Monitor: Detects fluctuations in data arrival rates.
- Statistics Aggregator: Collects processing statistics from worker nodes.
- Sampler: Utilizes advanced online spatial sampling methods (SAOS and ex-SAOS) to select representative data points.
- Controller: The brain of the system, which uses information from the monitor and statistics aggregator to calculate the optimal sampling rate based on the query's QoS goals. It employs statistical equations for accuracy control and a model-based approach for latency control.
- Approximator: Computes the final query results from the sampled data.
A key innovation is the controller's ability to learn from past data and adapt sampling strategies in real-time, ensuring a balance between providing fresh data (low latency) and retaining enough data for accurate analysis (high accuracy).
Datasets
The evaluation of ApproxSSPS used two real-world geo-referenced datasets:
- New York City taxicab trip data: Approximately 1.4 million records from January 2016, focusing on green taxi trips. The target variable for analysis was 'Trip-distance'.
- Shenzhen city electric taxi GPS mobility trips: Approximately 1.15 million records from a single day. The target variable was 'speed'.
These datasets represent typical mobility data found in smart city environments.
Key Insights and Ideas
- Spatial awareness is crucial: General-purpose sampling methods struggle with the inherent spatial characteristics of smart city data. ApproxSSPS's spatial sampling approach significantly improves accuracy by considering data co-locality.
- Dynamic QoS control is essential: Fixed sampling rates or reactive backpressure mechanisms are insufficient for highly dynamic data streams. A proactive, adaptive controller that learns and adjusts in real-time is key to maintaining both stability and performance.
- Trade-offs can be managed intelligently: The paper demonstrates that by dynamically adjusting sampling rates, a plausible balance can be struck between latency and accuracy, allowing systems to remain responsive even under heavy load.
- Simplicity in controllers yields efficiency: ApproxSSPS's lightweight QoS controllers add minimal overhead, preventing them from becoming a bottleneck themselves, unlike more complex baseline approaches.
Key Results
- Latency Control: ApproxSSPS effectively maintains system stability and meets latency targets (e.g., 1 second) even when arrival rates are five times greater than processing capacity. It achieves this by intelligently adjusting sampling rates, dropping less data than the baseline system under similar conditions.
- Accuracy Control: ApproxSSPS consistently outperforms the baseline system in terms of standard error across various sampling fractions and geohash precisions. For instance, it requires significantly less data (up to 98% less) to achieve stringent accuracy targets compared to the baseline.
- Overall Performance: The system demonstrates a significant improvement over a state-of-the-art baseline, achieving QoS goals with greater efficiency and stability.
Limitations and Future Work
- Limited Aggregate Functions: Currently, ApproxSSPS supports a limited set of predefined aggregate geo-statistics (count, average, sum). Future work aims to enable users to plug in custom aggregate functions for more flexibility.
- Cloud-Centricity: The current implementation is cloud-based. Future research plans to explore porting the sampling component to IoT devices to reduce cloud processing load.
- Temporal Dimension: While spatial awareness is addressed, the system has limited awareness of the temporal dimension. Future work will focus on incorporating spatiotemporal indexing and enabling sampling from multiple data streams for more complex spatiotemporal analytics.
- Advanced Analytics: The authors plan to extend ApproxSSPS to support more complex analytics, such as joining geo-referenced data streams (e.g., mobility data with meteorological data) to answer advanced queries.
Summary updated