Perona: Robust Infrastructure Fingerprinting for Resource-Efficient Big Data Analytics

Dominik Scheinert, Soeren Becker, Jonathan Bader, Lauritz Thamsen, Jonathan Will, Odej Kao

TL;DR

This paper introduces Perona, a novel method for "fingerprinting" big data analytics infrastructure. By using standardized benchmarks and advanced representation learning, Perona creates a compact, robust understanding of a system's capabilities that can be directly used to optimize resource configurations, leading to more efficient and cost-effective data processing.

What Problem Does This Paper Address?

Configuring resources for big data analytics, especially in dynamic cloud environments, is a significant challenge. Poor configurations lead to wasted resources, increased costs, and reduced performance. Existing automated approaches often rely on building performance models from past workload executions or iteratively profiling resources. However, these methods typically create an "implicit" understanding of the infrastructure that is highly specific to the tested workload and difficult to transfer to different workloads or environments. This means that valuable insights are lost, and the profiling process needs to be repeated for every new scenario, which is inefficient and costly. The paper aims to bridge this gap by providing an "explicit" and generalizable understanding of infrastructure capabilities.

Methods

Perona tackles this problem through a three-step process:

  1. Standardized Resource Benchmarking: The core idea is to use a consistent set of benchmarking tools with fixed configurations across different machines and infrastructures. This ensures that the collected performance metrics are directly comparable. The system collects a wide range of metrics and then applies preprocessing steps:
  2. Unification: Standardizing units for comparability.
  3. Selection: Filtering out metrics with low predictive value or insufficient historical variation.
  4. Orientation: Ensuring metrics are consistently oriented (e.g., always aiming to maximize or minimize). This results in a compact feature vector representing the benchmarked resource.

  5. End-to-End Contextual Representation Learning: Perona uses an autoencoder to learn a dense, low-dimensional representation of the benchmark metrics. This process effectively reduces dimensionality and captures the most relevant information. Crucially, it then employs a graph-based model to incorporate historical benchmark executions. This allows for context-aware outlier detection, identifying resource degradation or anomalies by comparing current benchmark behavior to past patterns.

  6. Aspect-Based Resource Ranking: To make the learned representations more useful, Perona introduces additional learning tasks:
  7. Clustering: Encouraging representations of the same benchmark type to be close together, while separating different types.
  8. Classification: Ensuring representations are informative enough to predict the benchmark type.
  9. Ranking: Enabling learned representations to be compared and ranked, typically based on a norm-based score. This allows for a fine-grained assessment of resource quality.

Datasets

The paper describes data acquisition and evaluation using several sources:

  • Internal Experiments: The authors conducted experiments on a K3s Kubernetes cluster in Google Cloud Platform (GCP). They used tools like sysbench, fio, ioping, qperf, and iperf3 for benchmarking. To simulate varying conditions, they sometimes injected stress using the ChaosMesh operator.
  • Publicly Available Datasets: Perona was evaluated in conjunction with existing resource configuration optimization methods (CherryPick, Arrow) using a publicly available dataset of 18 workloads running on 69 AWS configurations.
  • Related Work Datasets: The approach was also tested with methods from related work (Tarema, Lotaru) using their respective datasets and environments.

No new benchmarks were introduced; instead, Perona standardizes the usage of existing benchmarking tools to ensure comparability.

Key Insights and Ideas

  • Explicit vs. Implicit Infrastructure Understanding: The paper highlights the limitation of "implicit" infrastructure understanding derived from workload-specific profiling. Perona's "explicit" fingerprinting provides a generalizable model of the infrastructure itself.
  • Standardization is Key for Comparability: Using standardized benchmarking tools and configurations is crucial for creating comparable metrics across different machines and environments.
  • Representation Learning for Robustness: Autoencoders and graph-based models are powerful tools for extracting meaningful, low-dimensional representations from noisy benchmark data and for detecting anomalies.
  • Multi-Task Learning for Better Representations: Adding clustering, classification, and ranking objectives to the representation learning process significantly improves the quality and utility of the learned "fingerprints."
  • Context-Awareness for Degradation Detection: Incorporating historical benchmark data allows Perona to detect subtle resource degradation over time, which is critical for maintaining performance.

Key Results

  • High Accuracy in Representation Learning: The autoencoder achieved a Mean Squared Error (MSE) of 0.01 on test data, indicating effective reconstruction of input metrics.
  • Excellent Benchmark Type Classification: Learned representations achieved 100% accuracy in classifying benchmark types using a simple linear transformation, confirming they retain type-specific characteristics.
  • Strong Outlier Detection: Perona achieved an F1 score of 0.93 for the normal class and 0.75 for the outlier class in resource degradation detection, with a weighted accuracy of 90%.
  • Improved Resource Configuration: When integrated with existing methods like CherryPick and Arrow, Perona demonstrated improved cost-effectiveness in finding near-optimal cloud configurations, especially with more profiling runs.
  • Comparable Performance in Scientific Workflows: When used with Lotaru, Perona led to a median error increase of only 1.74% compared to the original method, while providing a more robust and generalizable fingerprint.

Limitations and Future Work

  • Data Limitations: The authors acknowledge that the amount of training data available was fairly limited, which might have impacted the outlier detection performance.
  • False Positives in Outlier Detection: While the F1 score for outliers is good, false positives can still occur. The authors suggest that these can be used as triggers for further investigation, which is a low-overhead process given the short duration of benchmarks.
  • Future Work: The authors plan to develop a unified framework for managing data processing jobs by integrating Perona with recent resource configuration optimization methods.

Summary updated

Abstract

Choosing a good resource configuration for big data analytics applications can be challenging, especially in cloud environments. Automated approaches are desirable as poor decisions can reduce performance and raise costs. The majority of existing automated approaches either build performance models from previous workload executions or conduct iterative resource configuration profiling until a near-optimal solution has been found. In doing so, they only obtain an implicit understanding of the underlying infrastructure, which is difficult to transfer to alternative infrastructures and, thus, profiling and modeling insights are not sustained beyond very specific situations. We present Perona, a novel approach to robust infrastructure fingerprinting for usage in the context of big data analytics. Perona employs common sets and configurations of benchmarking tools for target resources, so that resulting benchmark metrics are directly comparable and ranking is enabled. Insignificant benchmark metrics a red is carded by learning a low-dimensional representation of the input metric vector, and previous benchmark executions are taken into consideration for context-awareness as well, allowing to detect resource degradation. We evaluate our approach both on data gathered from our own experiments as well as within related works for resource configuration optimization, demonstrating that Perona captures the characteristics from benchmark runs in a compact manner and produces representations that can be used directly.
BibTeX
@inproceedings{scheinert2022perona,
  author = {Dominik Scheinert and Soeren Becker and Jonathan Bader and Lauritz Thamsen and Jonathan Will and Odej Kao},
  title = {{Perona: Robust Infrastructure Fingerprinting for Resource-Efficient Big Data Analytics}},
  booktitle = {IEEE Big Data},
  year = {2022},
  url = {https://openreview.net/forum?id=IO2SFPYVkS},
}
Figure