Efficient Solution for Large-Scale IoT Applications with Proactive Edge-Cloud Publish/Subscribe Brokers Clustering

Van-Nam Pham, Ga-Won Lee, Vandung Nguyen, Eui-Nam HuhSensors 2021 · DOI

TL;DR

This paper proposes a novel hierarchical Edge-Cloud publish/subscribe (pub/sub) broker system designed to efficiently handle the massive data streams from large-scale Internet of Things (IoT) applications. By proactively clustering edge brokers based on topic similarity and geographic proximity, and employing a two-tier routing scheme, the system significantly reduces data delivery latency and relay traffic, making it ideal for delay-sensitive IoT deployments.

What Problem Does This Paper Address?

The proliferation of IoT devices, generating vast amounts of data, poses a significant challenge for communication systems. Traditional cloud-centric approaches struggle with the low latency and high scalability required for real-time IoT applications. While edge computing offers a solution by processing data closer to the source, effectively managing and coordinating a large number of distributed edge brokers remains a complex problem. This paper tackles the challenge of designing a dynamic data delivery solution for large-scale IoT applications that can provide low latency and high scalability by intelligently organizing and routing data through a hierarchical Edge-Cloud pub/sub broker architecture.

Methods

The core of the proposed solution lies in a proactive clustering mechanism and a two-tier routing scheme.

  • Proactive Broker Clustering: The system employs implicit collaborative filtering (CF) techniques to predict future topics that edge brokers might subscribe to. This prediction is then used to cluster edge brokers based on their topic similarities and geographic proximity. This allows brokers within a cluster to efficiently exchange data directly.
  • Hierarchical Edge-Cloud Architecture: The system utilizes a two-tiered structure:
  • Edge Pub/Sub Brokers: Deployed in edge clouds, these brokers aggregate data from IoT gateways and provide local data storage and real-time processing. They also facilitate direct message delivery to IoT clients.
  • Cloud Pub/Sub Brokers: These brokers act as relays for data between edge broker clusters and for aggregating data from edge brokers for long-term storage and analysis.
  • Two-Tier Routing Scheme:
    • Intra-cluster: Brokers within the same cluster exchange data directly.
  • Inter-cluster: Cluster heads communicate with relay cloud brokers to facilitate data exchange between different clusters.
  • Coordination Module: A central coordinator manages broker clustering, topic channel bridging, and routing information updates, ensuring efficient system-wide data dissemination.

Datasets

The paper does not introduce new datasets. Instead, it uses simulated data based on:

  • Starbucks Store Locations: Approximately 3350 store locations were used as a coordinate pool for simulating the geographical distribution of pub/sub brokers.
  • Simulated Topic Subscriptions: Topic subscriptions were generated following Zipf's law to mimic realistic distribution patterns.

Key Insights and Ideas

  • Proactive Clustering is Key: Predicting future topic interests for brokers and clustering them accordingly is crucial for reducing latency. This moves beyond reactive approaches that only respond to existing subscriptions.
  • Edge-Cloud Synergy: The hierarchical model effectively leverages the strengths of both edge (low latency, local processing) and cloud (centralized management, large-scale storage) computing.
  • Topic Similarity as a Clustering Driver: Grouping brokers based on shared topic interests, not just location, significantly improves communication efficiency.
  • Balancing Latency and Scalability: The proposed clustering and routing strategy strikes a good balance between achieving low delivery latency and maintaining scalable broker node degrees, which is critical for large-scale deployments.

Key Results

The paper presents simulation results demonstrating the effectiveness of their proposed model:

  • Reduced Latency: The proposed model achieves significantly lower Average Delivery Latency (ADL) compared to the PubSubCoord-alike approach, with ADL being approximately 66% of the latter. For instance, with 500 brokers, the proposed ADL was 49.49 ms, while PubSubCoord-alike was 73.38 ms.
  • Lower Relay Traffic: The Average Forwarding Traffic (AFT) was also improved, with the proposal performing best, averaging about 7.77% less traffic than PubSubCoord-alike.
  • Scalable Node Degrees: The system maintains low broker node degrees even when scaling up the number of brokers and topic subscriptions, indicating efficient resource utilization.
  • Topic Correlation Benefits: Increasing topic correlations between brokers led to lower ADL and AFT, highlighting the advantage of clustering based on topic similarities.

Limitations and Future Work

  • Additional Cost Metrics: The authors acknowledge that while their simulation-based experiments validate the model, additional cost metrics like computing, storage, and networking costs need to be considered.
  • Emulated Edge-Cloud Testbed: Future work will involve developing the hierarchical broker model on an emulated Edge-Cloud testbed to provide a more realistic evaluation and to find the best cost-performance trade-off for wide-scale IoT applications.

Summary updated

BibTeX
@article{pham2021efficient,
  author = {Van-Nam Pham and Ga-Won Lee and Vandung Nguyen and Eui-Nam Huh},
  title = {{Efficient Solution for Large-Scale IoT Applications with Proactive Edge-Cloud Publish/Subscribe Brokers Clustering}},
  journal = {Sensors},
  year = {2021},
  doi = {10.3390/s21248232},
}
Figure