Abstract
Traditional visualization evaluation focuses on speed and accuracy of low-level tasks. This paper introduces a framework based on Bloom’s Taxonomy to measure understanding across six hierarchical levels. Through three case studies, the authors demonstrate that while design improvements significantly aid low-level data retrieval, they have a diminishing impact on high-level evaluation and decision-making.
Introduction: Beyond Low-Level Tasks
In the visualization community, "success" is often measured by performance on low-level perceptual tasks: How quickly can you find the value for Category X? or Which bar is taller? While these metrics are easy to quantify, they fail to capture the actual goal of most visualizations: fostering deep understanding, informing policy, or enabling predictions.
The authors argue that the field lacks a systematic way to evaluate these "higher-order" cognitive processes. Without a structured framework, designers cannot be certain if their choices—like unstacking a bar chart or fixing a non-chronological axis—actually help a reader synthesize information or make better decisions.
A New Framework: Adapting Bloom’s Taxonomy
The authors adapted Bloom’s Taxonomy, a well-known educational framework, into a six-level hierarchy for visualization evaluation:
- Knowledge: Locating and reporting specific data points.
- Comprehension: Summarizing the main message in one's own words.
- Application: Using the data to solve a simple problem (e.g., calculations).
- Analysis: Identifying trends or relationships between variables.
- Synthesis: Predicting future values or creating new insights.
- Evaluation: Making real-world arguments or policy recommendations based on the data.
Methodology and Case Studies
To test this framework, the authors conducted a study with 60 participants using three pairs of "Original" (confusing or misleading) vs. "Redesigned" (following best practices) visualizations.
The study utilized three real-world "case studies":
- Markets (The Economist): A chart with confusing double y-axes showing US trade deficits vs. manufacturing employment. The redesign used side-by-side charts.
- Immigration (The Globe and Mail): A stacked bar chart where the "time" variable was used in the legend rather than the x-axis. The redesign unstacked the bars.
- COVID-19 (Georgia Health Dept): A notorious retracted chart where the x-axis was sorted by value rather than date.
Key Insights and Results
The study yielded several critical insights into how design affects different levels of understanding:
- Non-Hierarchical Understanding: Unlike in education, where Bloom’s is strictly hierarchical, the authors found that visualization viewers might fail at low-level tasks (like finding a specific point) but still succeed at high-level ones (like identifying an overall trend).
- Design's Diminishing Returns: Redesigning a chart for clarity has a massive impact on Level 1 (Knowledge)—participants were significantly more likely to answer correctly with redesigned charts. However, it has a surprisingly small impact on Level 6 (Evaluation). People often arrived at the same policy conclusions regardless of whether the chart was "good" or "bad."
- The Resilience of the "Takeaway": Even with the misleading Georgia COVID chart, many participants still argued for "opening up" the state, citing a "downward trend" they perceived despite the visual chaos. This suggests that pre-existing biases can sometimes overpower the actual data encoding.
- Affordance of Patterns: The redesigned COVID chart allowed 60% of viewers to see a "bi-modal" (two-peak) pattern in the data, whereas only one person noticed it in the original version. This proves that design choices directly dictate which types of insights are even possible.
- Application Resilience: Surprisingly, there was no significant difference in accuracy for simple calculations (Level 3) between original and redesigned charts, suggesting that motivated users can overcome some poor design to "do the math."
Conclusion
The study demonstrates that while "good" design is critical for accurate data retrieval and trend analysis, it may not be enough to shift high-level evaluations or pre-existing beliefs. By using Bloom’s Taxonomy, visualization researchers can better understand where design interventions succeed and where cognitive biases take over. Future work should explore how numeracy skills and interactivity influence these levels of understanding.
Summary updated