VisQA: Quantifying Information Visualisation Recallability via Question Answering

Yao Wang, Chuhan Jiao, Mihai Bâce, Andreas Bulling

Abstract

Despite its importance for assessing the effectiveness of communicating information visually, fine-grained recallability of information visualisations has not been studied quantitatively so far. In this work, we propose a question-answering paradigm to study visualisation recallability and present VisRecall - a novel dataset consisting of 200 visualisations that are annotated with crowd-sourced human (N = 305) recallability scores obtained from 1,000 questions of five question types. Furthermore, we present the first computational method to predict recallability of different visualisation elements, such as the title or specific data values. We report detailed analyses of our method on VisRecall and demonstrate that it outperforms several baselines in overall recallability and FE-, F-, RV-, and U-question recallability. Our work makes fundamental contributions towards a new generation of methods to assist designers in optimising visualisations.

Introduction

In the field of data visualization, "memorability" has traditionally been measured by recognisability—whether a viewer remembers seeing a chart before. While useful, recognisability is a shallow metric that does not indicate if the viewer retained key insights or specific data points. This research addresses the gap in recallability, the ability to retrieve specific information from memory. By introducing a question-answering (QA) paradigm, the authors aim to transform the subjective assessment of visualization effectiveness into a quantitative, objective science.

Methodology

The authors developed VisRecall, a benchmark dataset containing 200 real-world visualizations and 1,000 expert-designed questions. These questions are categorized into five types: Title/Theme (T), Understanding structure/trends (U), Finding extrema (FE), Filtering (F), and Retrieving specific values (RV).

The human study involved 305 participants in a three-phase experiment designed to measure information retention over time.

The three-phase experimental pipeline: Encoding, Recall (with blurred visualization), and Recognition.

To automate this assessment, the researchers developed RecallNet, a computational model using an Xception backbone to predict both overall and fine-grained recallability scores for different visualization elements.

Key Findings and Results

A primary insight of the study is that Recognition ≠ Recall. While participants were highly accurate at recognizing charts they had seen (83%), their accuracy in answering specific questions about those charts was significantly lower (49%). This suggests that a visually memorable chart is not necessarily effective at communicating its data.

The study also quantified human performance across different tasks, finding that identifying the general theme (T) is the easiest task, while retrieving specific values (RV) is the most challenging.

Recallability scores across different question types, showing that general themes (T) are easier to recall than specific values (RV).

Additional findings include:

  • The 10-Second Rule: For most charts, 10 seconds of viewing was as effective as 20. Extra time only benefited complex, text-heavy visualizations.
  • The Pie Chart Paradox: Pie charts surprisingly exhibited higher recall scores compared to bar or line charts.
  • RecallNet Performance: The model outperformed standard baselines, achieving a Mean Squared Error (MSE) of 0.035 for overall recall and even lower errors for specific tasks like finding extrema (FE) and retrieving values (RV).

Conclusion

VisRecall and RecallNet provide the first computational framework for predicting the fine-grained recallability of visualization elements. While currently focused on static visualizations, this work paves the way for automated design assistants that can optimize charts based on the specific information a designer wants the audience to remember. Future work will explore interactive dashboards and integrate eye-tracking data to further refine these predictive models.

Summary updated

Abstract

Despite its importance for assessing the effectiveness of communicating information visually, fine-grained recallability of information visualisations has not been studied quantitatively so far. In this work, we propose a question-answering paradigm to study visualisation recallability and present VisRecall - a novel dataset consisting of 200 visualisations that are annotated with crowd-sourced human (N = 305) recallability scores obtained from 1,000 questions of five question types. Furthermore, we present the first computational method to predict recallability of different visualisation elements, such as the title or specific data values. We report detailed analyses of our method on VisRecall and demonstrate that it outperforms several baselines in overall recallability and FE-, F-, RV-, and U-question recallability. Our work makes fundamental contributions towards a new generation of methods to assist designers in optimising visualisations.
BibTeX
@article{wang2022visqa,
  author = {Yao Wang and Chuhan Jiao and Mihai Bâce and Andreas Bulling},
  title = {{VisQA: Quantifying Information Visualisation Recallability via Question Answering}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2022},
  url = {https://openreview.net/forum?id=uCPwguVCHx},
}

Loading related content...

Figure