TL;DR
This paper introduces a novel framework for evaluating data visualizations by adapting Bloom's taxonomy, a well-established educational tool. This method moves beyond simple accuracy and speed metrics to assess how visualizations facilitate deeper levels of understanding, from recalling facts to synthesizing new insights and evaluating information.
What Problem Does This Paper Address?
The authors tackle a critical challenge in the visualization community: the lack of comprehensive methods to evaluate how well a visualization truly helps users understand data. Current evaluation techniques often focus on low-level perceptual tasks like speed and accuracy, neglecting higher-order cognitive processes such as applying knowledge, analyzing complex relationships, synthesizing new ideas, and making informed judgments. This gap makes it difficult for designers to understand the true impact of their design choices on user comprehension and to create visualizations that effectively support complex decision-making.
Methods
The core of the paper is the adaptation of Bloom's six-level taxonomy of educational objectives (Knowledge, Comprehension, Application, Analysis, Synthesis, Evaluation) into a framework for evaluating data visualizations. For each level, the authors define specific tasks and provide example questions that can be posed to users interacting with a visualization. They then demonstrate this framework through three case studies, comparing original and redesigned versions of real-world visualizations. The evaluation involves collecting user responses to these questions and analyzing them to identify differences in understanding afforded by the different designs.
Datasets
The paper uses three pairs of real-world data visualizations as case studies:
- Markets: A visualization depicting the US trade deficit with China and US manufacturing employment.
- Immigration: A visualization showing immigrant populations in Thunder Bay, Ontario, and Canada by period of immigration.
- COVID-19: A visualization of COVID-19 cases over time in five counties (this chart was later retracted).
For each case study, the authors present both an original, potentially confusing visualization and a redesigned version intended to improve clarity.
Key Insights and Ideas
- Beyond Perception: Simple metrics like response speed and accuracy are insufficient for evaluating visualization effectiveness. Deeper cognitive levels are crucial for understanding and decision-making.
- Bloom's Taxonomy as a Lens: Adapting Bloom's taxonomy provides a structured way to probe different facets of understanding, from basic fact recall to complex evaluation.
- Design Affordances Matter: Different visualization designs can subtly or significantly influence what users notice, how they interpret trends, and what conclusions they draw.
- Hierarchical Structure Questioned: The study found limited evidence for a strict hierarchical relationship between Bloom's levels in the context of visualization understanding, suggesting that skills might be more independent than initially assumed.
- Unexpected Results: In some cases, redesigned visualizations didn't necessarily lead to better performance on all levels, highlighting the complexity of visualization design and user interpretation. For instance, one redesign didn't improve application-level performance, and in another, users drew similar conclusions despite design differences.
Key Results
The study used Amazon Mechanical Turk workers to evaluate the visualizations. Key findings include:
- Knowledge Level: Participants were significantly more accurate in answering knowledge-level questions (e.g., locating specific data points) when viewing redesigned visualizations compared to original ones (e.g., 21.45, p < 0.001 for COVID-19 chart).
- Comprehension Level: Redesigned COVID-19 charts led participants to discuss data at a county level more often. However, for the Immigration and Markets charts, there were no significant differences in comprehension-level responses between original and redesigned versions.
- Application Level: No significant differences in accuracy were found when participants were asked to calculate numerical differences between data points, suggesting this level might be less sensitive to design changes or more influenced by other factors like numeracy.
- Analysis Level: For the COVID-19 chart, participants viewing the redesigned version were more likely to notice the bi-modal trend (p = 0.0022). For the Immigration chart, participants were more likely to describe the trend as increasing with the original chart, an unexpected finding.
- Synthesis Level: Predictions for future values showed mixed results. For the Markets chart, participants predicted significantly higher trade deficits with the redesigned version (p = 0.027), but variances in predictions differed significantly.
- Evaluation Level: For the COVID-19 chart, participants in both groups were equally likely to argue for "opening up" based on perceived decreasing cases, despite design differences.
Limitations and Future Work
The authors acknowledge several limitations:
- Expert vs. Layperson Perspective: Designs created by domain experts might not reflect the affordances perceived by average users.
- Numeracy Skills: Performance on Knowledge, Application, and Analysis levels might be influenced by participants' numeracy skills, which were not explicitly measured.
- Time Span: Bloom's taxonomy was originally intended for longer learning spans, and its direct application to short visualization interactions might not perfectly capture all nuances.
- Textual vs. Interactive: Some levels, particularly Synthesis and Evaluation, might be better assessed through interactive tasks rather than purely textual responses.
Future work could involve:
- Developing more detailed example tasks for each level.
- Incorporating participant numeracy evaluation into experimental designs.
- Exploring how to better assess higher-level cognitive processes in interactive visualization contexts.
- Combining this method with other literacy measures (graphical, linguistic) for a more holistic evaluation.
- Investigating the potential for this framework to inform concrete design guidelines.
Summary updated