Abstract
Optimizing the performance of large-scale parallel codes is critical for efficient utilization of computing resources. Code developers often explore various execution parameters, such as hardware configurations, system software choices, and application parameters, and are interested in detecting and understanding bottlenecks in different executions. They often collect hierarchical performance profiles represented as call graphs, which combine performance metrics with their execution contexts. The crucial task of exploring multiple call graphs together is tedious and challenging because of the many structural differences in the execution contexts and significant variability in the collected performance metrics (e.g., execution runtime). In this paper, we present Ensemble CallFlow to support the exploration of ensembles of call graphs using new types of visualizations, analysis, graph operations, and features. We introduce ensemble-Sankey, a new visual design that combines the strengths of resource-flow (Sankey) and box-plot visualization techniques. Whereas the resource-flow visualization can easily and intuitively describe the graphical nature of the call graph, the box plots overlaid on the nodes of Sankey convey the performance variability within the ensemble. Our interactive visual interface provides linked views to help explore ensembles of call graphs, e.g., by facilitating the analysis of structural differences, and identifying similar or distinct call graphs. We demonstrate the effectiveness and usefulness of our design through case studies on large-scale parallel codes.
Introduction
In High-Performance Computing (HPC), optimization is a complex multi-variable problem. Developers frequently run simulations hundreds of times, varying parameters such as MPI libraries, hardware configurations, or grid sizes. Each execution generates a call graph—a hierarchical representation of function calls and their associated performance metrics (e.g., runtime or memory usage).
Comparing these graphs across a large ensemble is traditionally a manual and error-prone process. Existing tools often focus on single executions or simple side-by-side comparisons, which fail to scale when analyzing dozens or hundreds of runs. Furthermore, structural differences between call graphs—where certain functions are only invoked under specific parameters—make automated "diffing" difficult. Developers require a system that provides a high-level overview of performance variability while maintaining the ability to drill down into specific calling contexts.
The EnsembleCallFlow Framework
To address these challenges, we developed EnsembleCallFlow, a visual analytics tool designed for the comparative analysis of large call graph ensembles. The system introduces several key innovations in graph processing and visual encoding.
Graph Unification
A fundamental challenge in ensemble analysis is the structural variability between individual runs. EnsembleCallFlow addresses this through a "unify" operation. The system constructs a Super Graph that represents the union of all unique calling contexts found across the entire ensemble. When a specific function call is absent in a particular run, it is assigned a null value ($\emptyset$). This unification ensures a consistent visual layout, allowing users to compare performance across the ensemble without losing structural context.
Ensemble-Sankey Visualization
The core visual component is the ensemble-Sankey diagram. While standard Sankey diagrams use node height to represent a metric for a single execution, the ensemble-Sankey scales node height to the maximum value observed across the entire ensemble. Inside each node, ensemble gradients (color-mapped histograms) visualize the distribution of performance metrics across all runs. This approach makes "missingness" (structural differences) and performance outliers immediately visible as gaps or shifts in the gradient.
Case Studies and Results
We evaluated EnsembleCallFlow using three distinct HPC scenarios:
- AMM (Adaptive Multilinear Meshes): Analyzing 18 profiles of a C++ library revealed high variability in recursive functions. The tool helped developers identify significant overhead in
std::unordered_map, leading to the discovery of unnecessary hash collisions. - LULESH Hydrodynamics: By studying 8 runs ranging from 1 to 512 processes, we analyzed "weak scaling." The tool revealed that increasing core counts actually increased runtime for specific MPI functions—a bottleneck that would be obscured by simple averaging.
- OSU Bcast Benchmark: In an ensemble of 100 profiles collected over 24 hours, the tool successfully isolated 10 outlier runs. These were traced back to transient performance glitches in low-level messaging API calls (
psm_queue.c).
Conclusion
EnsembleCallFlow demonstrates that combining graph unification with distribution-aware visual encodings like the ensemble-Sankey can significantly accelerate the identification of performance bottlenecks in HPC workflows. Future work will focus on automated topology analysis to group runs by structural similarity and exploring temporal visualizations to track performance evolution across software version histories.
Summary updated