How to evaluate data visualizations across different levels of\n understanding

Alyxander Burns, Cindy Xiong, Steven Franconeri, Alberto Cairo, Narges Mahyar

TL;DR

This paper introduces a novel framework for evaluating data visualizations by adapting Bloom's taxonomy, a well-established educational tool. This method moves beyond simple accuracy and speed metrics to assess how visualizations facilitate deeper levels of understanding, from recalling facts to synthesizing new insights and evaluating information.

What Problem Does This Paper Address?

The authors tackle a critical challenge in the visualization community: the lack of comprehensive methods to evaluate how well a visualization truly helps users understand data. Current evaluation techniques often focus on low-level perceptual tasks like speed and accuracy, neglecting higher-order cognitive processes such as applying knowledge, analyzing complex relationships, synthesizing new ideas, and making informed judgments. This gap makes it difficult for designers to understand the true impact of their design choices on user comprehension and to create visualizations that effectively support complex decision-making.

Methods

The core of the paper is the adaptation of Bloom's six-level taxonomy of educational objectives (Knowledge, Comprehension, Application, Analysis, Synthesis, Evaluation) into a framework for evaluating data visualizations. For each level, the authors define specific tasks and provide example questions that can be posed to users interacting with a visualization. They then demonstrate this framework through three case studies, comparing original and redesigned versions of real-world visualizations. The evaluation involves collecting user responses to these questions and analyzing them to identify differences in understanding afforded by the different designs.

Datasets

The paper uses three pairs of real-world data visualizations as case studies:

  • Markets: A visualization depicting the US trade deficit with China and US manufacturing employment.
  • Immigration: A visualization showing immigrant populations in Thunder Bay, Ontario, and Canada by period of immigration.
  • COVID-19: A visualization of COVID-19 cases over time in five counties (this chart was later retracted).

For each case study, the authors present both an original, potentially confusing visualization and a redesigned version intended to improve clarity.

Key Insights and Ideas

  • Beyond Perception: Simple metrics like response speed and accuracy are insufficient for evaluating visualization effectiveness. Deeper cognitive levels are crucial for understanding and decision-making.
  • Bloom's Taxonomy as a Lens: Adapting Bloom's taxonomy provides a structured way to probe different facets of understanding, from basic fact recall to complex evaluation.
  • Design Affordances Matter: Different visualization designs can subtly or significantly influence what users notice, how they interpret trends, and what conclusions they draw.
  • Hierarchical Structure Questioned: The study found limited evidence for a strict hierarchical relationship between Bloom's levels in the context of visualization understanding, suggesting that skills might be more independent than initially assumed.
  • Unexpected Results: In some cases, redesigned visualizations didn't necessarily lead to better performance on all levels, highlighting the complexity of visualization design and user interpretation. For instance, one redesign didn't improve application-level performance, and in another, users drew similar conclusions despite design differences.

Key Results

The study used Amazon Mechanical Turk workers to evaluate the visualizations. Key findings include:

  • Knowledge Level: Participants were significantly more accurate in answering knowledge-level questions (e.g., locating specific data points) when viewing redesigned visualizations compared to original ones (e.g., 21.45, p < 0.001 for COVID-19 chart).
  • Comprehension Level: Redesigned COVID-19 charts led participants to discuss data at a county level more often. However, for the Immigration and Markets charts, there were no significant differences in comprehension-level responses between original and redesigned versions.
  • Application Level: No significant differences in accuracy were found when participants were asked to calculate numerical differences between data points, suggesting this level might be less sensitive to design changes or more influenced by other factors like numeracy.
  • Analysis Level: For the COVID-19 chart, participants viewing the redesigned version were more likely to notice the bi-modal trend (p = 0.0022). For the Immigration chart, participants were more likely to describe the trend as increasing with the original chart, an unexpected finding.
  • Synthesis Level: Predictions for future values showed mixed results. For the Markets chart, participants predicted significantly higher trade deficits with the redesigned version (p = 0.027), but variances in predictions differed significantly.
  • Evaluation Level: For the COVID-19 chart, participants in both groups were equally likely to argue for "opening up" based on perceived decreasing cases, despite design differences.

Limitations and Future Work

The authors acknowledge several limitations:

  • Expert vs. Layperson Perspective: Designs created by domain experts might not reflect the affordances perceived by average users.
  • Numeracy Skills: Performance on Knowledge, Application, and Analysis levels might be influenced by participants' numeracy skills, which were not explicitly measured.
  • Time Span: Bloom's taxonomy was originally intended for longer learning spans, and its direct application to short visualization interactions might not perfectly capture all nuances.
  • Textual vs. Interactive: Some levels, particularly Synthesis and Evaluation, might be better assessed through interactive tasks rather than purely textual responses.

Future work could involve:

  • Developing more detailed example tasks for each level.
  • Incorporating participant numeracy evaluation into experimental designs.
  • Exploring how to better assess higher-level cognitive processes in interactive visualization contexts.
  • Combining this method with other literacy measures (graphical, linguistic) for a more holistic evaluation.
  • Investigating the potential for this framework to inform concrete design guidelines.

Summary updated

Abstract

Understanding a visualization is a multi-level process. A reader must extract\nand extrapolate from numeric facts, understand how those facts apply to both\nthe context of the data and other potential contexts, and draw or evaluate\nconclusions from the data. A well-designed visualization should support each of\nthese levels of understanding. We diagnose levels of understanding of\nvisualized data by adapting Bloom's taxonomy, a common framework from the\neducation literature. We describe each level of the framework and provide\nexamples for how it can be applied to evaluate the efficacy of data\nvisualizations along six levels of knowledge acquisition - knowledge,\ncomprehension, application, analysis, synthesis, and evaluation. We present\nthree case studies showing that this framework expands on existing methods to\ncomprehensively measure how a visualization design facilitates a viewer's\nunderstanding of visualizations. Although Bloom's original taxonomy suggests a\nstrong hierarchical structure for some domains, we found few examples of\ndependent relationships between performance at different levels for our three\ncase studies. If this level-independence holds across new tested\nvisualizations, the taxonomy could serve to inspire more targeted evaluations\nof levels of understanding that are relevant to a communication goal.\n
BibTeX
@inproceedings{burns2020how,
  author = {Alyxander Burns and Cindy Xiong and Steven Franconeri and Alberto Cairo and Narges Mahyar},
  title = {{How to evaluate data visualizations across different levels of\textbackslash\{\}n understanding}},
  booktitle = {Beliv},
  year = {2020},
  doi = {10.1109/beliv51497.2020.00010},
}
Figure