S2CE: a hybrid cloud and edge orchestrator for mining exascale distributed streams

Nicolas Kourtellis, Herodotos Herodotou, Maciej Grzenda, Piotr Wawrzyniak, Albert Bifet

TL;DR

S2CE (Stream to Cloud & Edge) is a proposed hybrid orchestrator designed to bridge the growing gap between massive data generation at the edge and high-level analytics in the cloud. It provides a unified, vendor-agnostic framework that intelligently distributes machine learning tasks across edge and cloud resources to handle "exascale" data streams in real-time.

What Problem Does This Paper Address?

We are entering an era of "exascale" data, where billions of IoT devices, autonomous vehicles, and sensors generate data at a volume and velocity that far outpaces current centralized cloud processing capabilities.

The authors identify three critical bottlenecks in the status quo:

  1. The Latency Gap: Sending every byte of raw data to a central cloud for processing creates unacceptable delays for time-sensitive applications like self-driving cars or industrial automation.
  2. Vendor Lock-in: Most current big data analytics are tied to specific cloud providers (AWS, Azure, Google), making it difficult for companies to migrate workloads or use hybrid infrastructures.
  3. The Edge-Cloud Silo: Existing tools are usually optimized for either the cloud (high power, high latency) or the edge (low power, low latency), but rarely provide a seamless way to orchestrate tasks across both.

Methods

The paper proposes the S2CE architecture, a "first of its kind" multi-cloud and edge orchestrator. Unlike a standard processing engine, S2CE acts as a management layer that sits on top of existing Distributed Stream Processing Engines (DSPEs) like Apache Flink, Spark, and Storm.

Key architectural components include:

  • Intelligent Orchestration: A resource management module that decides whether to process data at the edge (to save bandwidth and reduce latency) or offload it to the cloud (for complex modeling) based on current network conditions and energy constraints.
  • Transformations & Data Fusion: A layer dedicated to cleaning, sampling, and merging heterogeneous data streams before they reach the ML models.
  • Adaptive Machine Learning: Support for incremental ML and self-adaptive Deep Learning (DL) models that can evolve as the data stream changes (addressing "concept drift").
  • Standardized APIs: S2CE uses "External" and "Internal" APIs to ensure interoperability between different cloud vendors and edge devices, effectively acting as a "universal translator" for data streams.

Datasets

As this is a vision and architecture paper, it does not present experimental results on specific external datasets. Instead, it highlights the necessity of synthetic stream generation. The authors argue that because real-world exascale data is often private or too massive to move, S2CE must include built-in generators to create realistic, privacy-preserving synthetic data for benchmarking and model training.

Key Insights and Ideas

  • Processing at the Source is Mandatory: The authors argue that decentralization isn't just a feature; it’s a requirement. Moving operators from the cloud to the edge is an NP-hard problem that requires automated orchestration rather than manual tuning.
  • The "Data Economy" Enabler: By providing standardized interconnection APIs, S2CE is designed to allow companies to not just process their own data, but to safely share or sell processed insights (e.g., a bus company selling traffic flow data to a city council) through a unified platform.
  • Unifying the ML Library: Currently, ML libraries for streaming are fragmented across different engines (e.g., Spark’s MLLib vs. FlinkML). S2CE aims to provide a comprehensive, unified library that works across all supported DSPEs.
  • Uncertainty Quantification: The paper emphasizes the need for Bayesian inference in stream mining to account for "noise" in sensor data and model uncertainty, which traditional deep networks often ignore.

Key Results

The primary "result" of this paper is a comprehensive feature comparison (Table 1) showing how S2CE fills the gaps left by industry leaders.

  • While tools like Apache Spark or Flink excel at distributed processing, they often lack built-in synthetic data generation, deep learning support for streams, or automated edge-cloud resource management.
  • S2CE is designed to be the "one-stop-shop" that combines these missing features into a single, easily configurable platform.

Limitations and Future Work

  • Implementation Complexity: While the architecture is well-defined, the authors acknowledge that building the "smart" algorithms for automated resource allocation across heterogeneous edge nodes remains a significant challenge.
  • Early Stages: The project is in its infancy. The author team has started bootstrapping the design using functionalities from Apache SAMOA, but a full-scale production deployment has not yet been demonstrated.
  • Security & Trust: While the paper mentions "distributed trust infrastructures," the specific mechanisms for securing data across multiple competing cloud providers and thousands of vulnerable edge devices require further exploration.

Summary updated

Abstract

The explosive increase in volume, velocity, variety, and veracity of data generated by distributed and heterogeneous nodes such as IoT and other devices, continuously challenge the state of art in big data processing platforms and mining techniques. Consequently, it reveals an urgent need to address the ever-growing gap between this expected exascale data generation and the extraction of insights from these data. To address this need, this position paper proposes Stream to Cloud & Edge (S2CE), a first of its kind, optimized, multi-cloud and edge orchestrator, easily configurable, scalable, and extensible. S2CE will enable machine and deep learning over voluminous and heterogeneous data streams running on hybrid cloud and edge settings, while offering the necessary functionalities for practical and scalable processing: data fusion and preprocessing, sampling and synthetic stream generation, cloud and edge smart resource management, and distributed processing.
BibTeX
@inproceedings{kourtellis2021s,
  author = {Nicolas Kourtellis and Herodotos Herodotou and Maciej Grzenda and Piotr Wawrzyniak and Albert Bifet},
  title = {{S2CE: a hybrid cloud and edge orchestrator for mining exascale distributed streams}},
  booktitle = {DEBS},
  year = {2021},
  doi = {10.1145/3465480.3466926},
}
Figure