Performance Visualization for TAU Instrumented Scientific Workflows

Cong Xie, Wei Xu, Sungsoo Ha, Kevin A. Huck, Sameer Shende, Hubertus Van Dam, Kerstin Kleese van Dam, Klaus MuellerVisigrapp (3: Ivapp) 2018 · DOI

TL;DR

This paper introduces a novel web-based visualization framework designed to help scientists understand and optimize the performance of complex scientific workflows. By integrating data from the TAU performance analysis tool and employing multi-level visualization techniques, the framework addresses the challenges of handling large, heterogeneous datasets generated by exascale scientific computing.

What Problem Does This Paper Address?

Modern scientific computing is rapidly scaling towards exascale, generating unprecedented amounts of data. This massive scale creates significant challenges in monitoring, evaluating, and improving the performance of scientific workflows, which are sequences of computational tasks. Traditional performance analysis tools often struggle to handle the sheer volume and complexity of data produced by these workflows, especially when aiming for real-time or near real-time insights. Specifically, existing tools lack the ability to directly capture and visualize the performance of entire workflows, which involve intricate interdependencies between various computational components. This gap hinders scientists' ability to identify bottlenecks, understand performance characteristics, and ultimately accelerate scientific discovery.

Methods

The authors propose a proof-of-concept framework built upon several key components:

  • Enhanced TAU Instrumentation: They improved the TAU (Tuning and Analysis Utilities) instrumentation method to specifically capture parallel workflow measurements, including function calls and MPI inter-process communication.
  • Web-Based Visualization Framework: A core contribution is a web-based framework that integrates data acquisition, storage, query, aggregation, and visualization. This allows for a unified view of performance data.
  • Multi-Level Visualization: The framework employs a "multilevel" visualization model, offering different levels of detail. This includes:
  • Overview Panel: A high-level summary of the entire workflow execution, showing timelines of trace events and message counts.
  • Trace Detailed View: A more granular view of function calls and message passing within a selected time range, with features like transparency and borderlines to enhance readability.
  • Node Detail View: A stacked bar graph representation for individual nodes, designed to handle dense function calls and reflect caller-callee relationships.
  • Profiles View: Stacked bar graphs visualizing performance metrics (like exclusive time) for each function per node/thread.
  • Data Aggregation: To manage large datasets, the framework uses data aggregation techniques. For the overview panel, this includes projecting events into time bins and aggregating message counts into histograms.
  • Algorithm for Event Matching: A novel algorithm (Algorithm 1) is presented to accurately pair function entry and exit events from trace data, enabling the reconstruction of call paths and call depths.

Datasets

The paper uses a chemistry workflow use case, specifically the NWChemEx toolkit, to demonstrate their framework. The workflow involves a molecular dynamics (MD) module and an analysis module. The MD simulation runs in parallel, emitting snapshots, while the analysis module is triggered by the production of expected data. For their demonstration, they used a classical MD run with 2200 timesteps on 4 nodes and a corresponding analysis with 1000 timesteps on 1 node. The MD execution took 309.3 seconds wall clock time. They selected the first 6.5 seconds of this execution for detailed visualization. The generated trace file was 16GB, and the profile file was 21MB, both in JSON format.

Key Insights and Ideas

  • Workflow-Centric Performance Analysis is Crucial: The paper highlights that traditional tools focusing on individual applications are insufficient for understanding complex scientific workflows. A unified view that connects component interactions is essential.
  • Multi-Level Detail is Key for Scalability: The proposed multi-level visualization approach, moving from an overview to detailed views, is critical for making massive datasets manageable and interpretable for users.
  • Visualizing Message Passing is Important: Beyond function calls, visualizing message passing between processes is vital for identifying communication bottlenecks within parallel workflows.
  • Web-Based Framework for Accessibility: A web-based approach makes the visualization tool accessible and facilitates collaboration, which is important for scientific teams.
  • Novel Event Matching Algorithm: The algorithm for pairing trace events is a key enabler for accurately reconstructing function call structures, especially in complex, nested call chains.

Key Results

The paper presents a functional proof-of-concept framework that successfully visualizes a scientific workflow. While specific quantitative performance improvements (e.g., speedup in analysis) are not the primary focus, the framework demonstrates its ability to:

  • Capture and integrate performance data from parallel workflows using enhanced TAU instrumentation.
  • Provide a linked display of trace and profile data across different levels of detail.
  • Visualize message communication patterns alongside function execution.
  • Handle large trace and profile files (16GB trace, 21MB profile) through its web-based architecture and aggregation techniques.

The paper showcases the framework's utility through figures illustrating the overview, trace detailed view, and node detail view, demonstrating how users can explore performance data from a high-level summary down to specific function calls and message exchanges.

Limitations and Future Work

The authors acknowledge several limitations and outline future work:

  • Scalability: While they have implemented level-of-detail mechanisms, further work is needed on the back-end for more robust data aggregation and efficient query/storage methods to truly handle exascale data.
  • Online Evaluation: The current framework is primarily designed for offline analysis. Extending it to support online data acquisition and processing is a significant future direction.
  • Workflow Component Data Flow: The paper mentions the need to visualize workflow component data flow, which is not fully addressed in the current work.
  • Additional Levels of Detail: Further development of visualization levels for both overview and detailed views is planned.
  • Extreme-Scale Use Cases: The authors intend to validate their framework on an extreme-scale or exascale use case and conduct more case studies.

Summary updated

BibTeX
@inproceedings{xu2018performance,
  author = {Wei (Celia) Xu and Cong Xie and Wei Xu and Sungsoo Ha and Kevin A. Huck and Sameer Shende and Hubertus Van Dam and Kerstin Kleese van Dam and Klaus Mueller},
  title = {{Performance Visualization for TAU Instrumented Scientific Workflows}},
  booktitle = {Visigrapp (3: Ivapp)},
  year = {2018},
  doi = {10.5220/0006646803330340},
}
Figure