Introduction
As smart cities evolve, they increasingly rely on crowdsensing—the practice of leveraging sensors in thousands of citizen smartphones to monitor urban dynamics like traffic flow, air quality, and public safety. However, this paradigm creates a massive "data deluge" that is often redundant, imprecise, and computationally expensive to process in real-time. Traditional stream processing frameworks often struggle with these noisy, high-volume flows because they lack the ability to prioritize data based on its importance or urgency.
This paper introduces Quasit, a distributed stream processing middleware designed specifically for the crowdsensing domain. Quasit allows developers to integrate fine-grained Quality of Service (QoS) requirements directly into their data pipelines, enabling systems to dynamically prioritize critical information and scale efficiently across large clusters.
The Crowdsensing Data Deluge
Existing stream processing tools, such as early versions of Apache S4 or MapReduce, were largely "QoS-blind." They treated all data packets with equal priority, making it difficult to prioritize urgent alerts over routine updates or to gracefully degrade performance when hardware resources became stretched.
To address this, the authors identify the need for a "crowdsensing loop" (illustrated above) that can balance the accuracy of sensing against the cost of processing. By incorporating quality requirements and user profiles into the task generation and data processing stages, the system can maintain high scalability without sacrificing the utility of the resulting models.
The Quasit Middleware Architecture
The Quasit architecture is built on several key technical components designed for high-performance concurrency and flexible orchestration:
- Streaming Information Graphs (SIGs): Developers model data flows as directed acyclic graphs where nodes represent "operators" (data processors) and edges represent communication channels.
- Fine-Grained QoS Policies: Quasit allows developers to annotate any element of the SIG with specific policies. These include Output Priority (e.g., prioritizing "Gold" tier services), Processing Caps (setting hard time limits on computations to prevent bottlenecks), and State Fault Tolerance (defining replicas for processor states).
- Master-Worker Architecture: A Quasit Domain Manager (QDM) handles orchestration and global admission control, while Quasit Runtime Nodes (QRNs) execute the actual processing tasks.
- High-Performance Tech Stack: The system is built using Scala for its Actor-based concurrency model and the Data Distribution Service (DDS) standard. By using DDS (which supports IP-multicast), Quasit avoids the overhead of managing thousands of point-to-point TCP connections, a common bottleneck in distributed frameworks.
Experimental Evaluation
The authors evaluated Quasit using a synthetic image manipulation pipeline consisting of 30 steps using OpenCV. The source data was a scaled version of a photo from the Kodak public test set, streamed periodically to simulate a mobile sensing feed.
Key Results
- Horizontal Scalability: Quasit demonstrated near-linear performance gains. A 4-node cluster achieved a 3.34x speedup in the "critical input rate" (the maximum data rate before overload) compared to a single node.
- Latency Advantage: Quasit significantly outperformed Apache S4. At a processing rate of 20 samples per second, Quasit maintained a Round Trip Time (RTT) of approximately 40 seconds, whereas S4’s latency spiked to over 60 seconds under the same load.
- Efficiency: Quasit’s threading model, which utilizes a work-stealing Fork/Join pool, proved more efficient than S4’s "one thread per stream" approach, which often suffers from high contention on multi-core systems.
Conclusion and Future Work
Quasit represents a significant step toward making smart city crowdsensing viable at scale by treating QoS as a first-class citizen in stream processing. One of the system's most effective innovations is the Processing Cap; the authors found that in crowdsensing, a result that is "too late" is often worse than one that is "slightly less precise." By interrupting slow computations to return cached values, Quasit keeps the entire pipeline moving.
Despite these successes, limitations remain. The Quasit Domain Manager currently represents a single point of failure, and the authors suggest a future hierarchical management structure to support thousands of nodes. Additionally, while the image pipeline provided a rigorous stress test, future work will involve validating the middleware in live, heterogeneous smart city deployments with actual mobile users.
Summary updated