ExtraPeak: Advanced Automatic Performance Modeling for HPC Applications

Alexandru Calotoiu, Marcin Copik, Torsten Hoefler, Marcus Ritter, Sergei Shudler, Felix Wolf, Felix Wolf 0001Software for Exascale Computing 2020 · DOI · DBLP

Abstract

As High-Performance Computing (HPC) systems approach the exascale era, the complexity of optimizing scientific applications grows exponentially. Manual performance modeling is often too slow and specialized for general use. The ExtraPeak project addresses this by introducing automated, multi-parameter performance modeling techniques. By leveraging hierarchical search heuristics and iterative refinement, the ExtraPeak toolset enables developers to identify scalability bottlenecks and predict resource requirements with high accuracy and minimal manual intervention.

Introduction

The transition to exascale computing brings a massive increase in the cost of code inefficiency. To optimize applications, developers must navigate a vast configuration space involving core counts, problem sizes, and various algorithmic parameters. Traditional trial-and-error experimentation is insufficient for identifying latent scalability issues. Analytical performance models—mathematical expressions of how performance scales—offer a solution, but their manual creation is a bottleneck. ExtraPeak aims to democratize performance analysis by automating the generation of these models.

Methodology

ExtraPeak builds upon Extra-P, an open-source tool that uses the Performance Model Normal Form (PMNF) to generate single-parameter models. The project introduces several key advancements to handle the complexities of modern HPC software.

Overview of the ExtraPeak project's base methods, advanced methods, and case studies.

Multi-Parameter Modeling

Modeling multiple variables simultaneously typically leads to a combinatorial explosion. ExtraPeak bypasses this by first generating single-parameter models and then using a hierarchical heuristic to evaluate only the most likely additive and multiplicative combinations.

Segmented and Iterative Modeling

Real-world performance often exhibits discontinuities due to hardware limits or algorithmic shifts. ExtraPeak includes algorithms to detect these change points through segmented modeling. Furthermore, an iterative search space refinement process prevents "overfitting" to system noise by only increasing model complexity when statistically justified.

Co-Design and Isoefficiency

The framework extends beyond execution time to model hardware requirements like memory bandwidth, FLOPs, and network traffic. It also introduces isoefficiency modeling for task-based programs to distinguish between hardware contention and inherent algorithmic limitations.

Evaluation and Results

The authors evaluated ExtraPeak using 100,000 synthetic test functions and a diverse set of real-world HPC applications, including Kripke, Sweep3D, and LULESH.

  • Efficiency: Heuristic search reduced modeling time from 107 hours (exhaustive search) to just 1.5 hours on synthetic data.
  • Accuracy: The heuristic approach identified the optimal model in 95.5% of cases and correctly identified the lead-order term in 100% of cases.
  • Search Space Reduction: For applications like BLAST and Kripke, the search space was reduced by five orders of magnitude.
  • Prediction Quality: Iterative refinement improved prediction accuracy significantly; for Sweep3D, the mean relative prediction error dropped from 17.2% to 6.3%.

Key Insights

The project demonstrates that "good enough" models across an entire codebase are more valuable than perfect models for isolated functions. By projecting multi-dimensional data onto single parameters, heuristics can effectively navigate complex search spaces. Additionally, iterative refinement serves as a robust defense against system noise, ensuring models remain simple and interpretable.

Limitations and Future Directions

A primary challenge remains the cost of data collection, which currently scales exponentially with the number of parameters. Future work focuses on:

  • Sparse Modeling: Using reinforcement learning to select a polynomial number of sample points, potentially reducing costs by 93%.
  • Taint Analysis: Utilizing the perf-taint tool to identify which parameters affect specific functions via static analysis, which has already shown a 45x reduction in instrumentation overhead for LULESH.

Summary updated

BibTeX
@inproceedings{calotoiu2020extrapeak,
  author = {Alexandru Calotoiu and Marcin Copik and Torsten Hoefler and Marcus Ritter and Sergei Shudler and Felix Wolf and Felix Wolf},
  title = {{ExtraPeak: Advanced Automatic Performance Modeling for HPC Applications}},
  booktitle = {Software for Exascale Computing},
  year = {2020},
  doi = {10.1007/978-3-030-47956-5\_15},
}
Figure