QoS-Aware Approximate Query Processing for Smart Cities Spatial Data Streams

Isam Mashhour Al Jawarneh, Paolo Bellavista, Antonio Corradi, Luca Foschini, Rebecca Montanari

TL;DR

This paper introduces ApproxSSPS, a novel system designed to efficiently process massive, real-time spatial data streams from smart cities. It tackles the challenge of fluctuating data rates and skewed distributions by employing QoS-aware approximate query processing, striking a balance between latency and accuracy to provide timely insights.

What Problem Does This Paper Address?

Smart cities generate enormous volumes of geo-referenced data from sources like IoT devices, particularly from mobile entities such as vehicles. This data is characterized by high skewness and fluctuating arrival rates, making it difficult for traditional data stream processing systems to keep up while meeting strict latency and accuracy requirements. Existing systems often resort to strategies like backpressure, which can lead to data loss and reduced accuracy, or over-provisioning resources, which is costly. There's a need for systems that can intelligently handle these dynamic data streams, ensuring timely and accurate results for critical smart city applications without compromising stability.

Methods

The core of this work is the ApproxSSPS system, built on Apache Spark Structured Streaming. It introduces a QoS controller that dynamically adjusts the sampling rate of incoming data to meet user-defined latency and/or accuracy targets. The system comprises five key components:

  • Monitor: Detects fluctuations in data arrival rates.
  • Statistics Aggregator: Collects processing statistics from worker nodes.
  • Sampler: Utilizes advanced online spatial sampling methods (SAOS and ex-SAOS) to select representative data points.
  • Controller: The brain of the system, which uses information from the monitor and statistics aggregator to calculate the optimal sampling rate based on the query's QoS goals. It employs statistical equations for accuracy control and a model-based approach for latency control.
  • Approximator: Computes the final query results from the sampled data.

A key innovation is the controller's ability to learn from past data and adapt sampling strategies in real-time, ensuring a balance between providing fresh data (low latency) and retaining enough data for accurate analysis (high accuracy).

Datasets

The evaluation of ApproxSSPS used two real-world geo-referenced datasets:

  • New York City taxicab trip data: Approximately 1.4 million records from January 2016, focusing on green taxi trips. The target variable for analysis was 'Trip-distance'.
  • Shenzhen city electric taxi GPS mobility trips: Approximately 1.15 million records from a single day. The target variable was 'speed'.

These datasets represent typical mobility data found in smart city environments.

Key Insights and Ideas

  • Spatial awareness is crucial: General-purpose sampling methods struggle with the inherent spatial characteristics of smart city data. ApproxSSPS's spatial sampling approach significantly improves accuracy by considering data co-locality.
  • Dynamic QoS control is essential: Fixed sampling rates or reactive backpressure mechanisms are insufficient for highly dynamic data streams. A proactive, adaptive controller that learns and adjusts in real-time is key to maintaining both stability and performance.
  • Trade-offs can be managed intelligently: The paper demonstrates that by dynamically adjusting sampling rates, a plausible balance can be struck between latency and accuracy, allowing systems to remain responsive even under heavy load.
  • Simplicity in controllers yields efficiency: ApproxSSPS's lightweight QoS controllers add minimal overhead, preventing them from becoming a bottleneck themselves, unlike more complex baseline approaches.

Key Results

  • Latency Control: ApproxSSPS effectively maintains system stability and meets latency targets (e.g., 1 second) even when arrival rates are five times greater than processing capacity. It achieves this by intelligently adjusting sampling rates, dropping less data than the baseline system under similar conditions.
  • Accuracy Control: ApproxSSPS consistently outperforms the baseline system in terms of standard error across various sampling fractions and geohash precisions. For instance, it requires significantly less data (up to 98% less) to achieve stringent accuracy targets compared to the baseline.
  • Overall Performance: The system demonstrates a significant improvement over a state-of-the-art baseline, achieving QoS goals with greater efficiency and stability.

Limitations and Future Work

  • Limited Aggregate Functions: Currently, ApproxSSPS supports a limited set of predefined aggregate geo-statistics (count, average, sum). Future work aims to enable users to plug in custom aggregate functions for more flexibility.
  • Cloud-Centricity: The current implementation is cloud-based. Future research plans to explore porting the sampling component to IoT devices to reduce cloud processing load.
  • Temporal Dimension: While spatial awareness is addressed, the system has limited awareness of the temporal dimension. Future work will focus on incorporating spatiotemporal indexing and enabling sampling from multiple data streams for more complex spatiotemporal analytics.
  • Advanced Analytics: The authors plan to extend ApproxSSPS to support more complex analytics, such as joining geo-referenced data streams (e.g., mobility data with meteorological data) to answer advanced queries.

Summary updated

Abstract

Large amounts of georeferenced data streams arrive daily to stream processing systems. This is attributable to the overabundance of affordable IoT devices. In addition, interested practitioners desire to exploit Internet of Things (IoT) data streams for strategic decision-making purposes. However, mobility data are highly skewed and their arrival rates fluctuate. This nature poses an extra challenge on data stream processing systems, which are required in order to achieve pre-specified latency and accuracy goals. In this paper, we propose ApproxSSPS, which is a system for approximate processing of geo-referenced mobility data, at scale with quality of service guarantees. We focus on stateful aggregations (e.g., means, counts) and top-N queries. ApproxSSPS features a controller that interactively learns the latency statistics and calculates proper sampling rates to meet latency or/and accuracy targets. An overarching trait of ApproxSSPS is its ability to strike a plausible balance between latency and accuracy targets. We evaluate ApproxSSPS on Apache Spark Structured Streaming with real mobility data. We also compared ApproxSSPS against a state-of-the-art online adaptive processing system. Our extensive experiments prove that ApproxSSPS can fulfill latency and accuracy targets with varying sets of parameter configurations and load intensities (i.e., transient peaks in data loads versus slow arriving streams). Moreover, our results show that ApproxSSPS outperforms the baseline counterpart by significant magnitudes. In short, ApproxSSPS is a novel spatial data stream processing system that can deliver real accurate results in a timely manner, by dynamically specifying the limits on data samples.
BibTeX
@inproceedings{jawarneh2021qos,
  author = {Isam Mashhour Al Jawarneh and Paolo Bellavista and Antonio Corradi and Luca Foschini and Rebecca Montanari},
  title = {{QoS-Aware Approximate Query Processing for Smart Cities Spatial Data Streams}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=rAwfO4CX8D},
}
Figure