Prediction of High-Performance Computing Input/Output Variability and Its Application to Optimization for System Configurations

Li Xu, Thomas Lux, Tyler H. Chang, Bo Li, Yili Hong, Layne T. Watson, Ali R. Butt, Danfeng Yao, Kirk W. CameronarXiv 2020 · DOI · arXiv

Abstract

Performance variability is an important measure for a reliable high performance computing (HPC) system. Performance variability is affected by complicated interactions between numerous factors, such as CPU frequency, the number of input/output (IO) threads, and the IO scheduler. In this paper, we focus on HPC IO variability. The prediction of HPC variability is a challenging problem in the engineering of HPC systems and there is little statistical work on this problem to date. Although there are many methods available in the computer experiment literature, the applicability of existing methods to HPC performance variability needs investigation, especially, when the objective is to predict performance variability both in interpolation and extrapolation settings. A data analytic framework is developed to model data collected from large-scale experiments. Various promising methods are used to build predictive models for the variability of HPC systems. We evaluate the performance of the methods by measuring prediction accuracy at previously unseen system configurations. We also discuss a methodology for optimizing system configurations that uses the estimated variability map. The findings from method comparisons and developed tool sets in this paper yield new insights into existing statistical methods and can be beneficial for the practice of HPC variability management.

Introduction

In the landscape of High-Performance Computing (HPC), raw throughput is often the primary metric of success. However, as systems scale toward the exascale, performance consistency—or the lack thereof—has emerged as a critical bottleneck. Performance variability refers to the run-to-run inconsistency in I/O speeds when executing the same task on identical hardware. When the standard deviation of performance reaches a significant fraction of the mean, it becomes nearly impossible for system engineers to determine if a configuration change resulted in a genuine improvement or a statistical fluke.

This research addresses the challenge of predicting and managing this variability. By treating the HPC system as a complex "black box," we develop a data-driven framework to build "variability maps." These maps allow for the prediction of system behavior across a vast parameter space, enabling the optimization of configurations to balance high performance with reliable stability.

Experimental Framework and Data Collection

To build robust predictive models, we conducted large-scale experiments using the IOzone benchmark on a Linux-based server. The study considered a comprehensive set of parameters across hardware, operating system, and application layers.

Summary of hardware, OS, and application variables used in the study.

The dataset comprised 94,770 total configuration combinations. Key variables included CPU clock frequency (15 levels), multiple I/O schedulers, various I/O operation modes, and scaling factors like the number of threads and file sizes. Of these, 94,185 points were utilized for training, while specific subsets were reserved to evaluate the models' abilities in both interpolation (predicting within the known range) and extrapolation (predicting outside the observed bounds, such as higher frequencies or thread counts).

Statistical Modeling Approach

We evaluated several statistical surrogate models to determine their efficacy in emulating HPC I/O variability, specifically the Performance Variability Measure (PVM), defined as the standard deviation of I/O throughput.

  • Standard Statistical Models: Multiple Linear Regression (LM) and Gamma Generalized Linear Models (GLM) served as baselines.
  • Spline and Distance-Based Methods: Multivariate Adaptive Regression Splines (MARS) and the Linear Shepard (LSP) method were tested for their ability to capture local non-linearities.
  • Machine Learning and Tree-Based Models: Bayesian Additive Regression Trees (BART) and Treed Gaussian Processes (TGP) were employed to handle complex interactions.
  • Categorical Gaussian Processes (CGP): These advanced models were specifically designed to handle the mixture of continuous variables (like frequency) and categorical variables (like scheduler types) inherent in HPC systems.

Key Insights: The Frequency Trap

One of the most significant findings is the relationship between raw performance and reliability. While increasing CPU frequency generally leads to higher throughput, it simultaneously triggers a dramatic spike in performance variability.

Relationship between CPU frequency, performance, and variability.

As shown in the experimental data, as frequency moves toward 3.0 GHz, the standard deviation (PVM) and the coefficient of variation (SD/Mean) increase sharply. This "Frequency Trap" suggests that the highest performance settings are often the least reliable, making variability prediction essential for system stability.

Results and Optimization

The comparative analysis of surrogate models yielded several key results:

  1. Interpolation Performance: For predicting within the bounds of the training data, MARS emerged as the top performer, followed closely by LSP and BART. These models effectively mapped the "landscape" of known configurations.
  2. Extrapolation Performance: When tasked with predicting "unseen" territory—such as performance at higher thread counts than those in the training set—CGP and LSP demonstrated superior accuracy. CGP, in particular, achieved the lowest error rates, making it a valuable tool for future hardware projections.
  3. The 16-Thread Sweet Spot: For read operations, performance peaked at 16 threads, matching the physical thread count of the test server. Pushing beyond this limit typically resulted in a significant spike in jitter.

Using these models, we applied the Augmented Lagrange method to solve a constrained optimization problem: identifying the system configuration that minimizes variability while maintaining a required throughput threshold. This approach successfully identified stable configurations that would be overlooked by traditional performance-only tuning.

Conclusion and Future Work

This study demonstrates that HPC I/O variability, while complex, is fundamentally predictable through statistical surrogate modeling. The developed framework provides a path for engineers to manage "jitter" in large-scale systems. Future research will focus on root cause analysis to understand the underlying hardware mechanisms driving these patterns and exploring security applications, such as using variability maps to detect the performance signatures of malware attacks.

Summary updated

Abstract

Performance variability is an important measure for a reliable high performance computing (HPC) system. Performance variability is affected by complicated interactions between numerous factors, such as CPU frequency, the number of input/output (IO) threads, and the IO scheduler. In this paper, we focus on HPC IO variability. The prediction of HPC variability is a challenging problem in the engineering of HPC systems and there is little statistical work on this problem to date. Although there are many methods available in the computer experiment literature, the applicability of existing methods to HPC performance variability needs investigation, especially, when the objective is to predict performance variability both in interpolation and extrapolation settings. A data analytic framework is developed to model data collected from large-scale experiments. Various promising methods are used to build predictive models for the variability of HPC systems. We evaluate the performance of the methods by measuring prediction accuracy at previously unseen system configurations. We also discuss a methodology for optimizing system configurations that uses the estimated variability map. The findings from method comparisons and developed tool sets in this paper yield new insights into existing statistical methods and can be beneficial for the practice of HPC variability management. This paper has supplementary materials online.
BibTeX
@misc{xu2020prediction,
  author = {Li Xu and Thomas Lux and Tyler H. Chang and Bo Li and Yili Hong and Layne T. Watson and Ali R. Butt and Danfeng Yao and Kirk W. Cameron},
  title = {{Prediction of High-Performance Computing Input/Output Variability and Its Application to Optimization for System Configurations}},
  year = {2020},
  eprint = {2012.07915},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.07915},
}
Figure