Designing an Interactive, Notebook-Embedded, Tree Visualization to Support Exploratory Performance Analysis

Connor Scully-Allison, Ian Lumsden, Katy Williams, Jesse Bartels, Michela Taufer, Stephanie Brink, Abhinav Bhatele, Olga Pearce, Katherine E. IsaacsarXiv 2022 · arXiv · OpenReview

Abstract

Interactive visualization via direct manipulation has inherent design trade-offs in flexibility, discoverability, and ease-of-use. Scripting languages can support a vast range of user queries and tasks, but may be more cumbersome for free-form exploration. Embedding interactive visualization in a scripting environment, such as a computational notebook, provides an opportunity for leveraging the strengths of both direct manipulation and scripting. We conduct a design study investigating this opportunity in the context of calling context trees as used for performance analysis of parallel software. Our collaborators make new performance analysis functionality available to users via Jupyter notebook examples, making the project setting conducive to such an investigation. Through a series of semi-structured interviews and regular meetings with project stakeholders, we produce a formal task analysis grounded in the expectation that tasks may be supported by scripting, interactive visualization, or both paradigms. We then design an interactive bivariate calling context tree visualization for embedding in Jupyter notebooks with features to pass data and state between the scripting and visualization contexts. We evaluated our embedded design with seven high performance computing experts. The experts were able to complete tasks and provided further feedback on the visualization and the notebook-embedded interactive visualization paradigm. We reflect upon the project and discuss factors in both the process and the design of the embedded visualization.

Introduction

Exploratory Data Analysis (EDA) often forces researchers into a frustrating trade-off between the infinite flexibility of scripting and the fluid, intuitive discovery of interactive visualization. In High-Performance Computing (HPC), analysts study Calling Context Trees (CCTs)—massive hierarchical structures representing program execution. These trees are often too large for manual inspection and too complex for simple scripts to filter effectively. This paper introduces a "scripting-aware" design framework for embedding interactive visualizations directly into computational notebooks, enabling a hybrid workflow that leverages the strengths of both paradigms.

A Scripting-Aware Task Analysis

To bridge the gap between code and UI, we established a conceptual framework for task suitability. We categorized tasks based on two primary dimensions: Task Specificity (complexity and concreteness) and Task Frequency.

Conceptual framework for task suitability based on frequency and specificity.

As illustrated above, high-frequency, low-specificity tasks—such as identifying an outlier or browsing a structure—are predominantly suited for interactive visualization. Conversely, low-frequency, high-specificity tasks—such as applying a complex regular expression filter across a library—are better handled via scripting. Many tasks fall into a middle ground where both paradigms can be used interchangeably or in tandem.

The Interactive CCT Visualization

Our solution centers on a bivariate calling context tree visualization designed specifically for the Jupyter notebook environment. The interface uses a node-link diagram to map two performance metrics simultaneously, allowing analysts to quickly identify "performance bugs" that might be small in code footprint but large in execution impact.

The interactive CCT visualization interface featuring bivariate encoding (b), legends (d), and histogram-based pruning (e).

Key features of the visualization include:

  • Bivariate Visual Encoding: Nodes use color to represent one metric (e.g., speedup) and size for another (e.g., total execution time). This helps users spot "bright red" (poor speedup) but "large" (high time) nodes.
  • Mass Pruning: A histogram widget (e) allows users to filter out thousands of insignificant nodes based on metric distributions, making massive datasets manageable.
  • Linked Views: A table view (c) provides precise data for selected nodes, while the tree view (a) maintains structural context.

Two-Way Data Synchronization

A core innovation of this work is the "live" state synchronization between the Python environment and the visual widget. Using a library called Roundtrip, we enabled two-way data transfer:

  1. Visualization to Script: When a user prunes a tree or selects a subset of data in the UI, the corresponding Python variable in the notebook updates automatically. This allows the visualization to act as a data transformer for subsequent code cells.
  2. Script to Visualization: Modifying data via a script cell immediately updates the visual representation, reducing the cognitive "temporal gap" between code execution and visual feedback.

Evaluation and Expert Insights

We evaluated the tool with seven HPC experts using real-world datasets from proxy applications like Lulesh and Kripke. Experts successfully identified performance bottlenecks and expressed a strong preference for the hybrid workflow. They specifically valued the ability to "debug" visual queries using small Python test cases and the efficiency gained from histogram-based pruning.

While users appreciated the automatic updates, the study also highlighted challenges. The non-linear nature of notebooks can still lead to "state confusion," where users lose track of which data version is currently displayed. Additionally, managing the screen real estate required for deep and wide CCTs within the horizontal constraints of a Jupyter cell remains an ongoing design challenge.

Conclusion

By treating visualization as an integrated component of the scripting pipeline rather than a static endpoint, we provide a more flexible environment for complex data exploration. This "scripting-aware" framework holds potential for other domains characterized by massive, complex data, such as Explainable AI (XAI) or genomics.

Summary updated

Abstract

Interactive visualization via direct manipulation has inherent design trade-offs in flexibility, discoverability, and ease-of-use. Scripting languages can support a vast range of user queries and tasks, but may be more cumbersome for free-form exploration. Embedding interactive visualization in a scripting environment, such as a computational notebook, provides an opportunity for leveraging the strengths of both direct manipulation and scripting. We conduct a design study investigating this opportunity in the context of calling context trees as used for performance analysis of parallel software. Our collaborators make new performance analysis functionality available to users via Jupyter notebook examples, making the project setting conducive to such an investigation. Through a series of semi-structured interviews and regular meetings with project stakeholders, we produce a formal task analysis grounded in the expectation that tasks may be supported by scripting, interactive visualization, or both paradigms. We then design an interactive bivariate calling context tree visualization for embedding in Jupyter notebooks with features to pass data and state between the scripting and visualization contexts. We evaluated our embedded design with seven high performance computing experts. The experts were able to complete tasks and provided further feedback on the visualization and the notebook-embedded interactive visualization paradigm. We reflect upon the project and discuss factors in both the process and the design of the embedded visualization.
BibTeX
@misc{scullyallison2022designing,
  author = {Connor Scully-Allison and Ian Lumsden and Katy Williams and Jesse Bartels and Michela Taufer and Stephanie Brink and Abhinav Bhatele and Olga Pearce and Katherine E. Isaacs},
  title = {{Designing an Interactive, Notebook-Embedded, Tree Visualization to Support Exploratory Performance Analysis}},
  year = {2022},
  eprint = {2205.04557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.04557},
}
Figure