TL;DR
Modern high-performance computing (HPC) workflows generate massive performance trace data, making traditional analysis impractical. Chimbuko introduces a novel, scalable, and distributed framework for real-time, in situ performance anomaly detection and analysis of these complex workflows, significantly reducing data volume while preserving crucial insights.
What Problem Does This Paper Address?
As HPC systems push towards exascale, scientific applications are increasingly adopting a workflow-based paradigm. These workflows, composed of numerous coupled tasks, introduce performance challenges beyond those of individual components, such as complex interdependencies and increased resource contention. The sheer volume of performance trace data generated by these workflows at scale (terabytes) makes traditional post-mortem analysis infeasible. Existing tools often struggle to capture and analyze this data in real-time, in situ, and at the workflow level, hindering the identification of performance bottlenecks and the optimization of these complex systems.
Methods
Chimbuko tackles this problem with a multi-pronged approach:
- Scalable Architecture: A distributed, in situ architecture minimizes data movement by processing performance trace data at the node where it's generated. This significantly reduces inter-node communication.
- Anomaly Detection (AD): Instead of processing all trace data, Chimbuko employs anomaly detection to filter out "normal" events and focus on those that indicate potential performance issues. This drastically reduces the data volume for human analysis. The AD module uses a statistical approach, flagging function calls with execution times significantly longer or shorter than the mean (controlled by a parameter $\alpha$).
- Prescriptive Provenance: Chimbuko extracts and reduces performance provenance data specifically for detected anomalies. This includes information about the execution stack, environment, and surrounding events, providing context for root cause analysis.
- In Situ Visualization: A coupled visualization framework allows for real-time monitoring of workflow anomalies. It offers multiscale views, dynamically updating trace anomalies at different levels of detail (e.g., rank, time frame, function call stack), enabling users to quickly identify and dive into potential issues.
- TAU Integration: The framework leverages the TAU Performance System for capturing trace and profile data, extending it with an ADIOS2 plugin for in situ data streaming.
Datasets
The paper primarily uses data generated from running scientific workflows on the Oak Ridge National Laboratory's Summit supercomputer. A specific case study involves a modified version of the NWChemEx computational chemistry code, simulating a biomolecular system with 1.2 million atoms. The data collected includes performance trace data from TAU, streamed via ADIOS2. No new benchmarks are introduced; rather, existing scientific workflows are used to demonstrate Chimbuko's capabilities.
Key Insights and Ideas
- Anomaly-driven data reduction is key: Focusing on anomalies, rather than all trace data, is crucial for making performance analysis of large-scale workflows tractable. This dramatically reduces data volume without losing critical information.
- In situ analysis is essential for exascale: Performing analysis concurrently with execution, without significant overhead, is a necessity for future HPC systems. Chimbuko's distributed, in situ design addresses this.
- Workflow-level analysis is a distinct challenge: Performance issues in workflows arise not just from individual components but from their interactions and shared resource usage. Chimbuko's ability to analyze at this level is a significant advancement.
- Prescriptive provenance provides context: Simply identifying an anomaly isn't enough; understanding why it occurred is vital. Prescriptive provenance, by linking anomalies to their execution context, facilitates root cause analysis.
- Scalable visualization is critical: Presenting complex, streaming performance data in an understandable and interactive way is paramount for user adoption and effective debugging. Chimbuko's multiscale visualization approach aims to achieve this.
Key Results
- Significant Data Reduction: Chimbuko achieved data reduction factors of 14x to 148x for filtered and unfiltered trace data, respectively, on large-scale runs (up to 2569 MPI ranks).
- Low Overhead: The addition of Chimbuko introduced a maximum overhead of about 8% on top of the overhead from TAU alone, demonstrating its efficiency.
- Comparable Accuracy: The distributed AD module showed comparable accuracy (97.6% on average) to a non-distributed version, while offering significantly better scalability.
- Effective Anomaly Detection: The case study successfully identified performance bottlenecks, such as communication delays and function execution anomalies, which were then investigated using the visualization tools. For instance, a specific function's launch delay was identified as a performance issue.
Limitations and Future Work
- Limitations:
- The current implementation of the visualization backend uses a simple, serverless SQL database (SQLite) which has limitations in handling concurrent writing operations.
- The anomaly detection currently focuses primarily on execution time imbalances.
- Future Work:
- Boost overall performance to further minimize application overhead and maximize detection power.
- Integrate more advanced AD algorithms.
- Seamlessly integrate the AD module and TAU for even lower overhead.
- Adopt a more advanced distributed database for simultaneous writing in the visualization component.
- Develop more advanced visual designs for the visualization.
Summary updated