Design Concerns for Integrated Scripting and Interactive Visualization in Notebook Environments

Connor Scully-Allison, Ian Lumsden, Katy Williams, Jesse Bartels, Michela Taufer, Stephanie Brink, Abhinav Bhatele, Olga Pearce, Katherine E. Isaacs

TL;DR

This paper explores how to best integrate interactive visualizations within computational notebooks for complex data analysis tasks, specifically in performance analysis. They propose a task analysis framework to guide design decisions and present a novel notebook-embedded visualization that bridges the gap between direct manipulation and scripting, enabling more fluid and insightful exploration.

What Problem Does This Paper Address?

Interactive visualizations are powerful for exploring data, but often limited to predefined tasks. Scripting offers flexibility but can be cumbersome for free-form exploration. Computational notebooks, like Jupyter, combine code and output, offering a promising environment for integrating these two paradigms. However, designing effective hybrid systems that seamlessly transfer information and state between visualization and scripting components remains a challenge. This paper tackles this by investigating how to design visualizations within notebooks that leverage the strengths of both direct manipulation and scripting, particularly for complex exploratory data analysis in fields like parallel performance analysis.

Methods

The authors employ a design study methodology, starting with a formal task analysis. They categorize tasks based on specificity (complexity/concreteness) and frequency, and also consider suitability for visualization or scripting. This analysis guides their design choices, assigning more frequent, less specific tasks to visualization and more specific, less frequent tasks to scripting. They then design and validate a notebook-embedded interactive visualization for calling context trees (CCTs), a common data structure in performance analysis. A key aspect of their design is enabling two-way data and state transfer between the visualization and Python scripting cells within the notebook. They also investigate different interaction modalities, specifically manual vs. automatic updating of cells.

Datasets

The paper uses datasets from the Kripke application, specifically a 64-core and a 128-core execution, resulting in trees of approximately 1500 and 2700 nodes, respectively. These datasets are used for evaluation tasks.

Key Insights and Ideas

  • Task-driven design is crucial: A systematic approach to analyzing tasks based on specificity, frequency, and suitability is essential for deciding where to place functionality (visualization vs. scripting).
  • Hybrid is better: For users familiar with scripting, a combined visualization-scripting approach offers significant advantages over purely visual or purely scripting methods, enabling complex queries and easier debugging.
  • Seamless state transfer is key: The ability for the visualization to update scripting cells and vice-versa, and for state to be easily saved and restored, is critical for a fluid workflow.
  • Automatic updating can reduce friction: While manual updating is more explicit, automatic updating can significantly streamline the workflow by reducing the need for manual re-execution of cells, especially in exploratory scenarios.
  • User experience matters: Even with powerful features, the design must be intuitive. Participants expressed a desire for features like undo and clearer state tracking.

Key Results

  • Participants successfully completed evaluation tasks using the designed visualization and workflow.
  • The task analysis framework proved effective in guiding design decisions.
  • Users expressed strong support for the integrated visualization-scripting approach, citing benefits like easier query construction, debugging, and sharing.
  • Automatic updating was generally preferred for exploratory tasks due to its fluidity, though concerns about state tracking were raised.
  • Participants found the linked table view (showing details of selected nodes) very useful for understanding and communicating findings.

Limitations and Future Work

The authors acknowledge limitations in the generalizability of their findings due to the small, specialized participant pool and the typical one-hour session duration. They suggest future work should focus on improving explicit state tracking for both notebook cells and data objects, and further exploring alternative designs for coordinated views within notebooks. They also note that the effectiveness of automatic updating might depend on users' familiarity with scripting and the complexity of the notebook environment.

Summary updated

Abstract

Interactive visualization can support fluid exploration but is often limited to predetermined tasks. Scripting can support a vast range of queries but may be more cumbersome for free-form exploration. Embedding interactive visualization in scripting environments, such as computational notebooks, provides an opportunity to leverage the strengths of both direct manipulation and scripting. We investigate interactive visualization design methodology, choices, and strategies under this paradigm through a design study of calling context trees used in performance analysis, a field which exemplifies typical exploratory data analysis workflows with Big Data and hard to define problems. We first produce a formal task analysis assigning tasks to graphical or scripting contexts based on their specificity, frequency, and suitability. We then design a notebook-embedded interactive visualization and validate it with intended users. In a follow-up study, we present participants with multiple graphical and scripting interaction modes to elicit feedback about notebook-embedded visualization design, finding consensus in support of the interaction model. We report and reflect on observations regarding the process and design implications for combining visualization and scripting in notebooks.
BibTeX
@article{scullyallison2024design,
  author = {Connor Scully-Allison and Ian Lumsden and Katy Williams and Jesse Bartels and Michela Taufer and Stephanie Brink and Abhinav Bhatele and Olga Pearce and Katherine E. Isaacs},
  title = {{Design Concerns for Integrated Scripting and Interactive Visualization in Notebook Environments}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2024},
  doi = {10.1109/tvcg.2024.3354561},
}
Figure