Abstract
As High-Performance Computing (HPC) systems approach the exascale era, the complexity of optimizing scientific applications grows exponentially. Manual performance modeling is often too slow and specialized for general use. The ExtraPeak project addresses this by introducing automated, multi-parameter performance modeling techniques. By leveraging hierarchical search heuristics and iterative refinement, the ExtraPeak toolset enables developers to identify scalability bottlenecks and predict resource requirements with high accuracy and minimal manual intervention.
Introduction
The transition to exascale computing brings a massive increase in the cost of code inefficiency. To optimize applications, developers must navigate a vast configuration space involving core counts, problem sizes, and various algorithmic parameters. Traditional trial-and-error experimentation is insufficient for identifying latent scalability issues. Analytical performance models—mathematical expressions of how performance scales—offer a solution, but their manual creation is a bottleneck. ExtraPeak aims to democratize performance analysis by automating the generation of these models.
Methodology
ExtraPeak builds upon Extra-P, an open-source tool that uses the Performance Model Normal Form (PMNF) to generate single-parameter models. The project introduces several key advancements to handle the complexities of modern HPC software.
Multi-Parameter Modeling
Modeling multiple variables simultaneously typically leads to a combinatorial explosion. ExtraPeak bypasses this by first generating single-parameter models and then using a hierarchical heuristic to evaluate only the most likely additive and multiplicative combinations.
Segmented and Iterative Modeling
Real-world performance often exhibits discontinuities due to hardware limits or algorithmic shifts. ExtraPeak includes algorithms to detect these change points through segmented modeling. Furthermore, an iterative search space refinement process prevents "overfitting" to system noise by only increasing model complexity when statistically justified.
Co-Design and Isoefficiency
The framework extends beyond execution time to model hardware requirements like memory bandwidth, FLOPs, and network traffic. It also introduces isoefficiency modeling for task-based programs to distinguish between hardware contention and inherent algorithmic limitations.
Evaluation and Results
The authors evaluated ExtraPeak using 100,000 synthetic test functions and a diverse set of real-world HPC applications, including Kripke, Sweep3D, and LULESH.
- Efficiency: Heuristic search reduced modeling time from 107 hours (exhaustive search) to just 1.5 hours on synthetic data.
- Accuracy: The heuristic approach identified the optimal model in 95.5% of cases and correctly identified the lead-order term in 100% of cases.
- Search Space Reduction: For applications like BLAST and Kripke, the search space was reduced by five orders of magnitude.
- Prediction Quality: Iterative refinement improved prediction accuracy significantly; for Sweep3D, the mean relative prediction error dropped from 17.2% to 6.3%.
Key Insights
The project demonstrates that "good enough" models across an entire codebase are more valuable than perfect models for isolated functions. By projecting multi-dimensional data onto single parameters, heuristics can effectively navigate complex search spaces. Additionally, iterative refinement serves as a robust defense against system noise, ensuring models remain simple and interpretable.
Limitations and Future Directions
A primary challenge remains the cost of data collection, which currently scales exponentially with the number of parameters. Future work focuses on:
- Sparse Modeling: Using reinforcement learning to select a polynomial number of sample points, potentially reducing costs by 93%.
- Taint Analysis: Utilizing the
perf-tainttool to identify which parameters affect specific functions via static analysis, which has already shown a 45x reduction in instrumentation overhead for LULESH.
Summary updated