Ruya: Memory-Aware Iterative Optimization of Cluster Configurations for Big Data Processing

Jonathan Will, Lauritz Thamsen, Jonathan Bader, Dominik Scheinert, Odej Kao

TL;DR

This paper introduces Ruya, a novel method that significantly speeds up the process of finding optimal cluster configurations for big data jobs. By intelligently profiling jobs to understand their memory needs, Ruya narrows down the search space, leading to over 50% fewer iterations compared to existing methods, thus reducing costs and improving efficiency.

What Problem Does This Paper Address?

Running big data processing jobs (like those on Apache Spark or Flink) on large clusters often involves a complex and costly process of selecting the right computational resources. Users, especially data engineers, struggle to pick the optimal combination of machine types and cluster sizes. This often leads to over-provisioning resources, resulting in unnecessarily high costs and underutilized hardware. A major culprit for this inefficiency is inadequate memory allocation, which can cause performance bottlenecks. While existing methods use iterative optimization (like Bayesian optimization) to find good configurations, they often overlook the crucial relationship between a job's memory requirements and the input data size, potentially leading to suboptimal choices and wasted search effort.

Methods

Ruya tackles this problem with a two-step, memory-aware approach:

  1. Memory Profiling and Extrapolation: Ruya first performs quick, resource-efficient profiling runs on small samples of the dataset, executed on a single machine. During these runs, it monitors the job's memory usage. This data is then used to extrapolate the memory requirement for the job when run on the full dataset. This step builds upon and extends the Crispy method for memory profiling.
  2. Prioritized Resource Configuration Search: Armed with the estimated memory needs, Ruya then intelligently explores the search space for cluster configurations. It prioritizes configurations that are likely to meet the job's memory requirements. This narrowed-down search space is then explored using a modified Bayesian optimization technique (similar to CherryPick). The search continues until an expected performance gain no longer justifies the cost of further exploration.

This memory-aware prioritization is the key innovation, as it significantly reduces the number of configurations that need to be evaluated.

Datasets

The evaluation of Ruya used an existing dataset introduced in a prior work (Arrow [13]). This dataset comprises 1031 Spark and Hadoop job executions that ran on 69 different AWS cluster configurations. The jobs involved seven different algorithms and were executed with two dataset sizes: "huge" and "bigdata." The cluster configurations varied in scale-out (4 to 48 machines) and machine types (c, m, and r classes in large, xlarge, and 2xlarge sizes). No new benchmarks were introduced; the paper leverages existing, well-defined workloads.

Key Insights and Ideas

  • Memory is King (for Bottlenecks): The paper strongly emphasizes that memory bottlenecks are a primary driver of cost and performance issues in big data processing, often more so than CPU or other resources.
  • Profiling is Cheap, Insight is Priceless: Performing short profiling runs on a single machine is a low-cost way to gain critical insights into a job's memory behavior, which can then be extrapolated to large-scale cluster environments.
  • Prioritization is the Key to Speed: By intelligently filtering the vast configuration space based on memory needs, Ruya dramatically accelerates the search for optimal settings. This is more effective than blindly exploring the entire space.
  • Adaptability to Data Size: Ruya's memory-aware approach makes it inherently adaptable to changes in input dataset size, a common scenario in real-world applications. Unlike methods that require re-profiling for every data size change, Ruya can adjust its search based on the learned memory-data size relationship.
  • Fallback Mechanism: When memory modeling proves difficult or inaccurate (e.g., for "unclear" job types), Ruya gracefully falls back to standard Bayesian optimization, ensuring it still provides a reasonable solution.

Key Results

  • Significant Iteration Reduction: Ruya reduces the number of search iterations required to find optimal or near-optimal configurations by over 50% compared to the baseline (CherryPick).
  • Faster Convergence: On average, Ruya finds the optimal resource configuration in about 12 iterations, compared to 24 iterations for the baseline. This advantage is even more pronounced for near-optimal configurations.
  • Low Profiling Overhead: The memory profiling phase for Ruya takes an average of just under 10 minutes on a personal computer, which is considered a negligible cost for the significant gains in optimization speed. This overhead is also independent of the full dataset size.
  • Effective for Linear Jobs: For jobs exhibiting a linear relationship between memory usage and dataset size, Ruya shows substantial improvements.
  • Robustness: Even in cases where memory modeling is challenging, Ruya performs comparably to or better than the baseline.

Limitations and Future Work

  • "Unclear" Job Types: For jobs where the memory usage pattern is neither clearly linear nor flat ("unclear"), Ruya's ability to reduce the search space is limited, and it relies more heavily on standard Bayesian optimization.
  • Extrapolation Accuracy: The effectiveness of Ruya hinges on the accuracy of its memory usage extrapolation from small samples to full datasets. While generally good, misclassifications can occur.
  • Future Work: The authors suggest future work could involve managing and sharing profiling and search knowledge across multiple jobs and iterations. This could enable more complex performance models that consider broader execution contexts, further enhancing resource selection. They also note that their approach is black-box and agnostic to specific dataflow frameworks, but could be combined with framework-specific optimizations (like Spark's SparkListener API) for even greater gains.

Summary updated

Abstract

Selecting appropriate computational resources for data processing jobs on large clusters is difficult, even for expert users like data engineers. Inadequate choices can result in vastly increased costs, without significantly improving performance. One crucial aspect of selecting an efficient resource configuration is avoiding memory bottlenecks. By knowing the required memory of a job in advance, the search space for an optimal resource configuration can be greatly reduced.Therefore, we present Ruya, a method for memory-aware optimization of data processing cluster configurations based on iteratively exploring a narrowed-down search space. First, we perform job profiling runs with small samples of the dataset on just a single machine to model the job’s memory usage patterns. Second, we prioritize cluster configurations with a suitable amount of total memory and within this reduced search space, we iteratively search for the best cluster configuration with Bayesian optimization. This search process stops once it converges on a configuration that is believed to be optimal for the given job. In our evaluation on a dataset with 1031 Spark and Hadoop jobs, we see a reduction of search iterations to find an optimal configuration by around half, compared to the baseline.
BibTeX
@inproceedings{will2022ruya,
  author = {Jonathan Will and Lauritz Thamsen and Jonathan Bader and Dominik Scheinert and Odej Kao},
  title = {{Ruya: Memory-Aware Iterative Optimization of Cluster Configurations for Big Data Processing}},
  booktitle = {IEEE Big Data},
  year = {2022},
  url = {https://openreview.net/forum?id=3XdjoXhG1e},
}
Figure