Analyzing Real Cluster Data for Formulating Allocation Algorithms in Cloud Platforms

Olivier Beaumont, Lionel Eyraud-Dubois, Juan Ángel Lorenzo del Castillo

TL;DR

This paper analyzes a massive Google cluster trace to understand real-world cloud workload characteristics. The authors identify a small subset of "dominant" jobs that consume most resources, enabling simplified modeling and more efficient resource allocation algorithms. They also reveal key insights into job dynamics, CPU-memory correlations, and machine failure patterns, crucial for designing robust cloud systems.

What Problem Does This Paper Address?

Designing effective resource allocation algorithms for cloud platforms is a significant challenge. A major hurdle is the scarcity of real-world, large-scale cloud usage data, often due to privacy concerns and the sheer size of commercial cloud infrastructures. This lack of data forces researchers to make many simplifying assumptions when developing and validating algorithms, which may not accurately reflect actual usage patterns. This paper aims to bridge this gap by providing a deep analysis of a publicly released Google cluster trace, focusing on understanding the characteristics of job resource usage, their dynamics, and interdependencies, to inform the design of more realistic and efficient allocation strategies.

Methods

The authors employ a data-driven approach, analyzing a large Google cluster trace. Their methodology involves:

  • Identifying "Dominant Jobs": They define dominant jobs as the smallest set of jobs that collectively account for at least 90% of the CPU usage at any given time. This subset is then analyzed to reduce the complexity of the workload.
  • Static and Dynamic Analysis: They examine both the static properties (e.g., resource usage distribution) and dynamic behaviors (e.g., how resource usage changes over time, job lifespans) of these dominant jobs.
  • Correlation Analysis: They investigate correlations between different resource dimensions (CPU and memory) within jobs and also look for temporal patterns (hourly, daily, weekly) in resource usage.
  • Failure Analysis: They analyze machine removal events to understand failure patterns and test assumptions about their independence.
  • Model Formalization: Based on their findings, they propose a simplified way to formalize the resource allocation problem, aiming for a tractable model with a small set of parameters.

Datasets

The primary dataset used in this paper is the Google cluster trace, which was made publicly available. This trace contains detailed resource usage information (CPU, memory) for jobs and tasks running on a large-scale production cluster over a 29-day period, collected at 5-minute intervals. The authors do not introduce new benchmarks but rather analyze and extract insights from this existing, massive dataset.

Key Insights and Ideas

  • The "80/20" Rule in Cloud: A surprisingly small fraction of jobs (less than 6%) account for a vast majority (around 95%) of CPU usage and 90% of memory usage. This "dominant job" concept is crucial for simplifying allocation algorithms.
  • CPU-Memory Independence is Common: For a significant portion of jobs (over 70%), memory usage is largely independent of CPU usage, simplifying multi-dimensional packing problems. For another substantial portion (26.5%), memory usage is affine with CPU usage.
  • Job Priorities Matter for Resource Usage: Dominant jobs are heavily skewed towards certain priority classes, particularly "Normal Production" and "Gratis" jobs, which influences their scheduling and eviction behavior.
  • Predictable Patterns Exist: Many long-running jobs exhibit daily and even weekly patterns in their CPU usage, suggesting opportunities for predictive resource allocation.
  • Machine Failures Aren't Always Independent: While Poisson distribution is a common assumption for machine failures, the trace data suggests correlations, especially during planned maintenance events, which need to be considered for robust fault tolerance.

Key Results

  • Dominant Jobs: The set of dominant jobs, defined as those accounting for at least 90% of CPU usage, represents only 3.8% of all jobs in the trace. On average, about 240 jobs (5.8% of running jobs) account for 94.7% of CPU and 89% of memory usage per time window.
  • CPU-Memory Correlation: Over 72.5% of analyzed jobs show memory usage that is independent of CPU usage, and an additional 26.5% show an affine relationship. This significantly simplifies resource allocation modeling.
  • Job Durations: "Normal Production" jobs are significantly longer-lived, with half running for over 31.7 hours, while "Gratis" and "Other" priority jobs are much shorter-lived.
  • Temporal Patterns: Over half of the jobs exhibit strong daily patterns, and two-thirds have significant daily patterns. Weekly patterns are significant for about half of the jobs.
  • Machine Failures: The trace shows a higher frequency of time windows with multiple machine removals than predicted by a Poisson distribution, indicating potential correlations in failures.

Limitations and Future Work

  • Maintenance vs. Failure Distinction: The paper acknowledges that differentiating between planned maintenance events and actual machine failures is difficult with the provided trace, which limits the accuracy of failure analysis.
  • Model Completeness: While the paper proposes a simplified model, it notes that a complete generating model for all parameters would be valuable.
  • Algorithmic Design: The authors plan to use their findings to design and validate efficient resource allocation algorithms that account for the observed dynamicity and patterns.
  • Failure Characterization: A more in-depth analysis of machine failures, distinguishing between maintenance and actual failures, is suggested for future work.
  • Parameter Generation: Developing a complete model for generating parameters that capture the workload characteristics is an open area for future research.

Summary updated

Abstract

A problem commonly faced in Computer Science research is the lack of real usage data that can be used for the validation of algorithms. This situation is particularly true and crucial in Cloud Computing. The privacy of data managed by commercial Cloud infrastructures, together with their massive scale, make them very uncommon to be available to the research community. Due to their scale, when designing resource allocation algorithms for Cloud infrastructures, many assumptions must be made in order to make the problem tractable. This paper provides deep analysis of a cluster data trace recently released by Google and focuses on a number of questions which have not been addressed in previous studies. In particular, we describe the characteristics of job resource usage in terms of dynamics (how it varies with time), of correlation between jobs (identify daily and/or weekly patterns), and correlation inside jobs between the different resources (dependence of memory usage on CPU usage). From this analysis, we propose a way to formalize the allocation problem on such platforms, which encompasses most job features from the trace with a small set of parameters.
BibTeX
@inproceedings{beaumont2014analyzing,
  author = {Olivier Beaumont and Lionel Eyraud-Dubois and Juan Ángel Lorenzo del Castillo},
  title = {{Analyzing Real Cluster Data for Formulating Allocation Algorithms in Cloud Platforms}},
  booktitle = {SBAC-PAD},
  year = {2014},
  url = {https://openreview.net/forum?id=Bu0n6wchDg},
}
Figure