Sponge: Fast Reactive Scaling for Stream Processing with Serverless Frameworks

Won Wook Song, Taegeon Um, Sameh Elnikety, Myeongjae Jeon, Byung-Gon ChunUSENIX ATC 2023 · OpenReview

TL;DR

This paper introduces Sponge, a novel stream processing system that tackles the challenge of handling unpredictable bursts of data in real-time by intelligently combining traditional Virtual Machines (VMs) with cost-effective Serverless Functions (SFs). Sponge achieves low latency and reduced costs by dynamically offloading bursty workloads to SFs, while leveraging VMs for stable loads, outperforming existing scaling methods.

What Problem Does This Paper Address?

Real-time stream processing systems are crucial for applications that need to analyze continuously generated data, like financial markets or IoT sensor networks. A major challenge in these systems is dealing with sudden, unpredictable spikes in data volume (bursty loads). Existing systems struggle to maintain low latency (meeting Service Level Objectives or SLOs) during these bursts without incurring significant costs, often by over-provisioning resources that remain idle most of the time. Traditional approaches to scaling on demand, like migrating tasks and their associated state to new machines, are often too slow or introduce substantial overheads, especially when dealing with large amounts of state. This paper addresses the need for a system that can react quickly and cost-effectively to these unpredictable load fluctuations while guaranteeing low latency.

Methods

Sponge's core innovation lies in its hybrid approach, combining VMs and SFs, and its novel techniques for managing this integration:

  • Hybrid Cloud Resource Utilization: Sponge uses VMs for stable, continuous workloads due to their lower operational cost. When a burst of traffic occurs, it rapidly spins up SF instances, which have a much faster startup time (hundreds of milliseconds vs. tens of seconds for VMs), to handle the increased load.
  • Redirect-and-Merge Mechanism: To avoid the costly state migration from VMs to SFs, Sponge introduces a "redirect-and-merge" strategy. Incoming bursty data is quickly redirected to SF instances. For stateful operators, SFs maintain partial states, which are then merged back into the original states on the VMs. This significantly reduces the overhead associated with state transfer.
  • Fast Reactive Scaling: Sponge continuously monitors CPU usage and input rates to detect bottlenecks. It employs a dynamic offloading policy that precisely determines which tasks to offload and how many SF instances are needed to meet latency SLOs, aiming for a CPU utilization range of 60-80% to balance efficiency and responsiveness.
  • Compile-time Graph Rewriting: To minimize runtime overhead, Sponge transforms the application's Directed Acyclic Graph (DAG) at compile time. This involves inserting new "proxy" operators:
  • Router Operators (ROs): These manage data communication between VM and SF instances, especially for shuffle or broadcast operations.
  • Transient Operators (TOs): Cloned versions of original operators designed to run on SFs, ready to receive events immediately.
  • Merge Operators (MOs): These facilitate merging partial states from SFs back to the original VM states for stateful operators.
  • Cold Start Latency Reduction: Sponge employs strategies like pre-warming SF workers and utilizing AWS SnapStart to further minimize the startup latency of SF instances.

Datasets

The paper evaluates Sponge using the NEXMark benchmark. NEXMark is a widely used benchmark for stream processing that simulates auctions and bid data streams, featuring diverse queries with complex dataflow and stateful operations. The authors selected 6 out of 8 NEXMark queries (Q1, Q4, Q5, Q6, Q7, Q8) to represent different characteristics like statefulness, data communication patterns, and windowing operations. They also generated various bursty input patterns (sudden increase, sine curve, gradual increase) to simulate realistic load fluctuations.

Key Insights and Ideas

  • SFs are not a silver bullet for all stream processing challenges: While SFs offer fast startup, their limitations (like restricted inter-SF communication) and higher per-unit cost mean they need to be integrated intelligently, not as a complete replacement for VMs.
  • State migration is the primary bottleneck for scaling stateful stream processing: The paper highlights that the overhead of moving large states between machines is a major impediment to achieving low-latency scaling. Sponge's redirect-and-merge approach directly addresses this by avoiding full state migration.
  • Compile-time optimization is crucial for runtime performance: By transforming the DAG and inserting proxy operators before runtime, Sponge avoids expensive re-optimization during scaling events, leading to faster reactions.
  • Hybrid resource management is key to balancing cost and performance: The combination of cheap, stable VMs for baseline loads and fast, elastic SFs for bursts offers a compelling solution for both low latency and cost efficiency.

Key Results

  • Sponge reduces 99th-percentile tail latencies by an average of 88% compared to VM-based scaling (VMBase) and 70% compared to SF-based scaling without Sponge's optimizations (SFBase).
  • Sponge performs comparably to over-provisioned VM solutions (Over), which are significantly more expensive.
  • Sponge effectively handles 5x burstiness in input rates, with performance clearly distinguishable from baselines under such loads.
  • The redirect-and-merge mechanism significantly reduces latency, especially for stateful queries, by avoiding state migration overheads.
  • The router operators (ROs) are effective in scenarios where VMs become bottlenecks due to shuffle communication, reducing latency by up to 70%.
  • Transient operators (TOs) enable faster redirection of tasks, reducing latency by up to 28% compared to migration-based approaches.
  • Merge operators (MOs) are critical for stateful queries, reducing latencies by up to 56% and 38% in complex queries (Q5, Q8) compared to SpongeTO without MOs.
  • Sponge's cost is smaller than over-provisioned VM solutions when bursty loads occur for short durations (less than 15% of the time).

Limitations and Future Work

  • VM initialization overhead: While SFs are fast, the initial setup of VMs for stable loads still incurs a significant startup delay (25-30 seconds), which is a limitation for scenarios requiring immediate VM scaling.
  • Cost of SFs for persistent bursts: If bursty loads persist for extended periods, the cost of using SF instances can exceed that of statically over-provisioning VMs. Sponge addresses this by offering a mechanism to transition to new VMs if bursts are prolonged.
  • Future Work: The authors suggest extending Sponge to handle more complex data communication patterns and exploring more sophisticated scaling policies that can adapt to a wider range of workload characteristics. They also mention the potential for further optimization of state management and inter-operator communication within the SF environment.

Summary updated

BibTeX
@inproceedings{song2023sponge,
  author = {Won Wook Song and Taegeon Um and Sameh Elnikety and Myeongjae Jeon and Byung-Gon Chun},
  title = {{Sponge: Fast Reactive Scaling for Stream Processing with Serverless Frameworks}},
  booktitle = {USENIX ATC},
  year = {2023},
  url = {https://openreview.net/forum?id=KUABqIjVZc},
}
Figure