C3O: Collaborative Cluster Configuration Optimization for Distributed Data Processing in Public Clouds

Jonathan Will, Lauritz Thamsen, Dominik Scheinert, Jonathan Bader, Odej Kao

TL;DR

This paper introduces C3O, a system that optimizes cloud cluster configurations for distributed data processing by leveraging shared historical runtime data from multiple users. C3O uses collaborative runtime prediction models to achieve mean absolute errors below 3% on Spark jobs, significantly improving upon traditional single-user approaches.

What Problem Does This Paper Address?

Running large-scale data processing jobs on distributed dataflow systems (like Apache Spark or Flink) in public clouds is powerful but complex. Choosing the right cluster configuration – specifically, the type of virtual machines and the number of nodes (scale-out) – is a significant challenge. Incorrect choices can lead to either performance bottlenecks (jobs run too slowly) or wasted resources (paying for idle capacity). This is especially difficult for users who don't frequently run such jobs or lack dedicated operations staff. Existing solutions often involve costly trial-and-error or rely on limited, single-user data, which doesn't capture the full spectrum of job behaviors and contexts.

Methods

C3O tackles this by building a collaborative system for runtime prediction and cluster configuration. The core idea is to pool shared historical runtime data from various users and their diverse job executions. This data is used to train specialized regression models that predict job runtimes for different cluster configurations.

Key methodological aspects include:

  • Collaborative Runtime Prediction: Instead of relying solely on a user's own past job runs, C3O leverages a shared repository of runtime data. This allows models to learn from a much wider range of contexts, parameters, and dataset characteristics.
  • Optimistic Models: The paper proposes "optimistic" runtime models that assume pairwise independence of runtime-influencing features. These models break down the prediction into two parts: a Scale-out to Speedup Model (SSM) and an Inputs Behavior Model (IBM). This approach helps manage the "curse of dimensionality" in high-feature spaces.
  • Dynamic Model Selection: C3O doesn't rely on a single model. It uses cross-validation to dynamically select the most accurate prediction model for a given job and context, adapting to varying amounts and quality of training data.
  • Two-Stage Configuration: The cluster configuration process is split into two stages: first, selecting the optimal machine type (often job-dependent), and second, determining the appropriate scale-out based on user-defined runtime targets and cost preferences.

Datasets

The evaluation used a dataset comprising 930 unique runtime experiments across 5 different Spark jobs. These jobs were executed on Amazon EMR, using Hadoop and Spark. The experiments were compiled with Scala version 2.12.8. Each of the 930 experiments was run five times, and median runtimes were used. The dataset includes common jobs like Sort, Grep, SGD, K-Means, and PageRank. The paper notes that three features are shared across all jobs (machine type, scale-out, dataset size/problem size), with additional context-specific features for each job.

Key Insights and Ideas

  • The Power of Collaboration: The most significant insight is that sharing runtime data across users dramatically improves prediction accuracy. This collaborative approach allows models to generalize better and handle diverse execution contexts, leading to more reliable predictions than single-user models.
  • Context Matters: Runtime is not just about dataset size and scale-out. Algorithm parameters and specific dataset characteristics (e.g., number of unique pages in a PageRank dataset) have a substantial impact, and collaborative data captures this variability.
  • Dynamic Model Selection is Crucial: No single model is universally best. The effectiveness of a model changes with the amount and characteristics of the training data. C3O's ability to dynamically select the best model based on cross-validation is key to its robust performance.
  • Early Stages of Training Data are Critical: Some models, like the BOM, struggle with very little training data (less than 10 points) because their underlying assumptions (e.g., needing at least two data points with varying scale-out but identical other features for SSM) are not met. This highlights the importance of having a sufficient initial dataset.

Key Results

  • C3O achieved mean absolute errors below 3% in its experimental evaluation with 930 unique Spark jobs.
  • In collaborative scenarios (using global training data), C3O exhibited substantially lower errors compared to the baseline approach Ernest, with mean absolute percentage errors ranging from 2.61% to 6.22% compared to Ernest's 5.82% to 14.04%.
  • For most jobs, C3O's predictor was at least as accurate as its most accurate constituent model, and in other cases, it was within half a percent of the best individual model.
  • The GBM model showed an increase in accuracy in the collaborative scenario despite learning more features, indicating its ability to benefit from larger, more diverse datasets.

Limitations and Future Work

  • Limited Benefit for Highly Customized Jobs: The collaborative approach primarily benefits commonly occurring jobs. Users with highly customized, user-specific jobs might still need to rely on their own local runtime data.
  • Future Work: The authors plan to work on quick but effective profiling methods to assist users with customized jobs. They also aim to further explore the challenges and possibilities of collaborative approaches.

Summary updated

Abstract

Distributed dataflow systems enable data-parallel processing of large datasets on clusters. Public cloud providers offer a large variety and quantity of resources that can be used for such clusters. Yet, selecting appropriate cloud resources for dataflow jobs - that neither lead to bottlenecks nor to low resource utilization - is often challenging, even for expert users such as data engineers. We present C3O, a collaborative system for optimizing data processing cluster configurations in public clouds based on shared historical runtime data. The shared data is utilized for predicting the runtimes of data processing jobs on different possible cluster configurations, using specialized regression models. These models take the diverse execution contexts of different users into account and exhibit mean absolute errors below 3% in our experimental evaluation with 930 unique Spark jobs.
BibTeX
@inproceedings{will2021c,
  author = {Jonathan Will and Lauritz Thamsen and Dominik Scheinert and Jonathan Bader and Odej Kao},
  title = {{C3O: Collaborative Cluster Configuration Optimization for Distributed Data Processing in Public Clouds}},
  booktitle = {IC2E},
  year = {2021},
  url = {https://openreview.net/forum?id=cWaSV2kP8d},
}
Figure