Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation

Dominik Scheinert, Houkun Zhu, Lauritz Thamsen, Morgan K. Geldenhuys, Jonathan Will, Alexander Acker, Odej Kao

TL;DR

This paper introduces Enel, a novel dynamic scaling approach for distributed dataflow jobs that leverages graph propagation on attributed job graphs. By considering job context and task statistics, Enel accurately predicts job runtimes and adapts resource allocation to meet performance targets, even in the presence of failures.

What Problem Does This Paper Address?

Modern data processing relies heavily on distributed dataflow systems like Spark and Flink to handle massive datasets. While initial resource allocation can be guided by runtime predictions, the actual performance of these jobs is highly dynamic, influenced by factors like data locality, inter-task dependencies, and hardware failures. Manually configuring clusters for optimal performance is challenging and time-consuming. Existing solutions often struggle to adapt to these dynamic changes or require specialized models for each job, limiting reusability. This paper addresses the need for an automated, context-aware, and robust dynamic scaling mechanism that can continuously optimize resource allocation to meet user-defined runtime targets, even in the face of unpredictable events.

Methods

Enel's core innovation lies in its graph-based modeling approach. It represents a dataflow job as a directed, attributed graph where nodes represent tasks and edges represent dependencies. Each node is enriched with "descriptive properties" that capture the execution context (e.g., dataset size, hardware specs) and runtime statistics.

The key steps are:

  • Context Encoding: Descriptive properties are transformed into fixed-size vectors. This involves using hashing for textual properties and binary representation for numerical ones. These vectors are then further processed by an auto-encoder to create dense "embeddings."
  • Prediction and Propagation: Enel uses a graph neural network (GNN) to predict node-level runtimes and metrics. This involves:
  • Predicting node "overhead" (e.g., due to I/O changes) using a function f1.
  • Predicting node runtime using a function f2 that incorporates overhead and end scale-out.
  • Accumulating predicted runtimes across the graph to estimate the total job runtime.
  • Crucially, Enel predicts metrics for nodes where actual metrics might not be available by learning relationships between predecessor nodes and their metrics. This is achieved using an attention mechanism (inspired by [33]) to weigh the influence of predecessors.
  • Inter-Iteration Graph Relationships: For iterative dataflow jobs, Enel establishes relationships between consecutive iteration graphs using "summary nodes." These nodes encapsulate information from previous iterations (scale-out, context, and metric vectors) and are used as predecessors for the next iteration's graph, aiding in prediction stabilization.

Enel is designed as a "black-box" approach, meaning it doesn't make assumptions about the underlying dataflow system or resource manager, promoting reusability.

Datasets

The paper uses four benchmark Spark jobs: Logistic Regression (LR), Multilayer Perceptron Classifier (MPC), K-Means, and Gradient Boosted Trees (GBT). These jobs were evaluated using three synthetically generated datasets:

  • Multiclass: A classification dataset with 3 classes and 200 features.
  • Points: A two-dimensional dataset with points sampled from a Gaussian Mixture Model.
  • Vandermonde: A regression dataset with 19 features, generated using a custom generator to compute the Vandermonde matrix.

The paper notes that for initial training, they rely on historical data from previous runs of these jobs.

Key Insights and Ideas

  • Graph Structure is Key: Explicitly modeling the dataflow job as a graph, rather than treating tasks independently, is crucial for accurate prediction and scaling. The dependencies and flow of information within the graph are vital.
  • Context is King (for Reusability): By encoding rich descriptive properties of the execution context, Enel achieves a reusable model. This means a single trained model can be applied across different iterations of an iterative job or even across different jobs with similar contexts, avoiding the need for job-specific model retraining.
  • Propagation for Robustness: The message propagation mechanism allows Enel to propagate predictions and metrics through the graph. This helps in identifying potential bottlenecks and adapting scaling decisions proactively, even when faced with unexpected events like node failures.
  • Handling Failures Gracefully: Enel demonstrates a notable stability in the presence of failures, particularly for K-Means. The ability to retrain and fine-tune the model periodically allows it to correct prediction errors and adapt to the new reality of the execution environment.
  • Iterative Jobs are a Sweet Spot: The iterative nature of many dataflow jobs provides a natural source of multiple graphs (one per iteration) for training and fine-tuning Enel's model, making it particularly well-suited for such workloads.

Key Results

  • Improved Prediction Performance: Enel's predictions generally stabilize over time, with the exception of occasional outliers. The paper reports mean and median values for CVC (runtime constraint violation count) and CVS (runtime constraint violation sum) across different job types and run ranges, showing that predicted runtimes are often slightly off and regularly below the target runtime.
  • Robustness to Failures: Enel shows better robustness to failures compared to the baseline method (Ellis [21]), especially for MPC and K-Means. While Ellis struggles to react to failures, Enel's predictions are more appropriate.
  • Comparable to State-of-the-Art: The authors conclude that Enel achieves results comparable to state-of-the-art solutions, particularly given its single-model approach and occasional retraining.
  • Efficient Inference: The inference (prediction) process is described as fast and robust.
  • Training Time: Fine-tuning the Enel model takes a few seconds, which is considered acceptable given that dataflow jobs typically run for much longer. The time required depends on the number of graphs to process, with more complex jobs like GBT taking longer.

Limitations and Future Work

  • Initial Volatility: Enel's prediction performance can be volatile initially, especially for jobs like MPC and K-Means, due to limited training data.
  • Occasional Outliers: Despite overall stabilization, single outliers in predictions can still occur.
  • Manual Retraining: While the model is reusable, fine-tuning and retraining are still performed periodically, not fully automated in real-time.

Future work directions include:

  • Exploiting Graph Information Further: Investigating more advanced ways to leverage the dataflow graph for optimized resource allocation, dynamic scaling, and scheduling of multiple dataflow jobs.
  • Multi-Job Scheduling: Developing solutions for common scenarios in shared clusters involving the management of multiple dataflow jobs concurrently.

Summary updated

Abstract

Distributed dataflow systems like Spark and Flink enable the use of clusters for scalable data analytics. While runtime prediction models can be used to initially select appropriate cluster resources given target runtimes, the actual runtime performance of dataflow jobs depends on several factors and varies over time. Yet, in many situations, dynamic scaling can be used to meet formulated runtime targets despite significant performance variance.This paper presents Enel, a novel dynamic scaling approach that uses message propagation on an attributed graph to model dataflow jobs and, thus, allows for deriving effective rescaling decisions. For this, Enel incorporates descriptive properties that capture the respective execution context, considers statistics from individual dataflow tasks, and propagates predictions through the job graph to eventually find an optimized new scale-out. Our evaluation of Enel with four iterative Spark jobs shows that our approach is able to identify effective rescaling actions, reacting for instance to node failures, and can be reused across different execution contexts.
BibTeX
@inproceedings{scheinert2021enel,
  author = {Dominik Scheinert and Houkun Zhu and Lauritz Thamsen and Morgan K. Geldenhuys and Jonathan Will and Alexander Acker and Odej Kao},
  title = {{Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation}},
  booktitle = {IPCCC},
  year = {2021},
  url = {https://openreview.net/forum?id=7303RB7nk9},
}
Figure