How to evaluate data visualizations across different levels of understanding

Cindy Xiong Bearfield, Alyxander Burns, Cindy Xiong, Steven Franconeri, Alberto Cairo, Narges Mahyar

Abstract

Traditional visualization evaluation focuses on speed and accuracy of low-level tasks. This paper introduces a framework based on Bloom’s Taxonomy to measure understanding across six hierarchical levels. Through three case studies, the authors demonstrate that while design improvements significantly aid low-level data retrieval, they have a diminishing impact on high-level evaluation and decision-making.

Introduction: Beyond Low-Level Tasks

In the visualization community, "success" is often measured by performance on low-level perceptual tasks: How quickly can you find the value for Category X? or Which bar is taller? While these metrics are easy to quantify, they fail to capture the actual goal of most visualizations: fostering deep understanding, informing policy, or enabling predictions.

The authors argue that the field lacks a systematic way to evaluate these "higher-order" cognitive processes. Without a structured framework, designers cannot be certain if their choices—like unstacking a bar chart or fixing a non-chronological axis—actually help a reader synthesize information or make better decisions.

A New Framework: Adapting Bloom’s Taxonomy

The authors adapted Bloom’s Taxonomy, a well-known educational framework, into a six-level hierarchy for visualization evaluation:

  1. Knowledge: Locating and reporting specific data points.
  2. Comprehension: Summarizing the main message in one's own words.
  3. Application: Using the data to solve a simple problem (e.g., calculations).
  4. Analysis: Identifying trends or relationships between variables.
  5. Synthesis: Predicting future values or creating new insights.
  6. Evaluation: Making real-world arguments or policy recommendations based on the data.

Methodology and Case Studies

To test this framework, the authors conducted a study with 60 participants using three pairs of "Original" (confusing or misleading) vs. "Redesigned" (following best practices) visualizations.

The COVID-19 case study: The original chart (left) scrambled dates to show a false downward trend, while the redesign (right) restored chronological order.

The study utilized three real-world "case studies":

  • Markets (The Economist): A chart with confusing double y-axes showing US trade deficits vs. manufacturing employment. The redesign used side-by-side charts.
  • Immigration (The Globe and Mail): A stacked bar chart where the "time" variable was used in the legend rather than the x-axis. The redesign unstacked the bars.
  • COVID-19 (Georgia Health Dept): A notorious retracted chart where the x-axis was sorted by value rather than date.
Comparison of original and redesigned charts for the Markets and Immigration case studies.

Key Insights and Results

The study yielded several critical insights into how design affects different levels of understanding:

  • Non-Hierarchical Understanding: Unlike in education, where Bloom’s is strictly hierarchical, the authors found that visualization viewers might fail at low-level tasks (like finding a specific point) but still succeed at high-level ones (like identifying an overall trend).
  • Design's Diminishing Returns: Redesigning a chart for clarity has a massive impact on Level 1 (Knowledge)—participants were significantly more likely to answer correctly with redesigned charts. However, it has a surprisingly small impact on Level 6 (Evaluation). People often arrived at the same policy conclusions regardless of whether the chart was "good" or "bad."
  • The Resilience of the "Takeaway": Even with the misleading Georgia COVID chart, many participants still argued for "opening up" the state, citing a "downward trend" they perceived despite the visual chaos. This suggests that pre-existing biases can sometimes overpower the actual data encoding.
  • Affordance of Patterns: The redesigned COVID chart allowed 60% of viewers to see a "bi-modal" (two-peak) pattern in the data, whereas only one person noticed it in the original version. This proves that design choices directly dictate which types of insights are even possible.
  • Application Resilience: Surprisingly, there was no significant difference in accuracy for simple calculations (Level 3) between original and redesigned charts, suggesting that motivated users can overcome some poor design to "do the math."

Conclusion

The study demonstrates that while "good" design is critical for accurate data retrieval and trend analysis, it may not be enough to shift high-level evaluations or pre-existing beliefs. By using Bloom’s Taxonomy, visualization researchers can better understand where design interventions succeed and where cognitive biases take over. Future work should explore how numeracy skills and interactivity influence these levels of understanding.

Summary updated

Abstract

Understanding a visualization is a multi-level process. A reader must extract\nand extrapolate from numeric facts, understand how those facts apply to both\nthe context of the data and other potential contexts, and draw or evaluate\nconclusions from the data. A well-designed visualization should support each of\nthese levels of understanding. We diagnose levels of understanding of\nvisualized data by adapting Bloom's taxonomy, a common framework from the\neducation literature. We describe each level of the framework and provide\nexamples for how it can be applied to evaluate the efficacy of data\nvisualizations along six levels of knowledge acquisition - knowledge,\ncomprehension, application, analysis, synthesis, and evaluation. We present\nthree case studies showing that this framework expands on existing methods to\ncomprehensively measure how a visualization design facilitates a viewer's\nunderstanding of visualizations. Although Bloom's original taxonomy suggests a\nstrong hierarchical structure for some domains, we found few examples of\ndependent relationships between performance at different levels for our three\ncase studies. If this level-independence holds across new tested\nvisualizations, the taxonomy could serve to inspire more targeted evaluations\nof levels of understanding that are relevant to a communication goal.\n
BibTeX
@inproceedings{bearfield2020how,
  author = {Cindy Xiong Bearfield and Alyxander Burns and Cindy Xiong and Steven Franconeri and Alberto Cairo and Narges Mahyar},
  title = {{How to evaluate data visualizations across different levels of understanding}},
  booktitle = {Beliv},
  year = {2020},
  doi = {10.1109/beliv51497.2020.00010},
}
Figure