VisRecall: Quantifying Information Visualisation Recallability via Question Answering

Yao Wang, Chuhan Jiao, Mihai Bâce, Andreas Bulling

TL;DR

This paper introduces VisRecall, a novel dataset and framework for quantitatively measuring how well people remember specific details from information visualizations. They propose a question-answering approach to assess recallability and develop a computational model that can predict it, paving the way for tools that help designers create more effective visualizations.

What Problem Does This Paper Address?

Effectively communicating information through visualizations is crucial, but understanding what people actually remember from them has been challenging. Previous methods for assessing "recallability" (what viewers remember) were subjective, time-consuming, and relied on expert judgment, providing only a single score for the entire visualization. This left a gap in understanding which specific elements or aspects of a visualization contribute to better recall, and how to computationally predict this fine-grained recallability.

Methods

The authors propose a question-answering (QA) paradigm, similar to Visual Question Answering (VQA) in computer vision, to quantify visualization recallability. Instead of building a model to answer questions, they use human performance on QA tasks as a measure of recall. They designed five types of questions to capture different aspects of recall:

  • T-questions: About the title or general theme.
  • U-questions: About understanding structure or trends.
  • FE-questions: Finding extreme values.
  • F-questions: Filtering elements based on criteria.
  • RV-questions: Retrieving specific values.

They then developed RecallNet, a convolutional neural network (CNN) model, to predict both overall and fine-grained recallability scores. RecallNet leverages the Xception architecture, pre-trained on ImageNet, to encode visual information and outputs two scores: one for general recallability and one for fine-grained recallability.

Datasets

The primary contribution is the VisRecall dataset, which consists of:

  • 200 real-world information visualizations.
  • 1,000 questions across the five defined types.
  • Crowd-sourced human recallability scores from 305 participants.

This dataset is novel because it focuses on fine-grained recallability and uses expert-generated questions, differentiating it from existing chart QA datasets which often use synthesized data and simpler question types. They also collected recognisability scores for comparison.

Key Insights and Ideas

  • QA is a powerful tool for recallability: Using a question-answering framework provides a more objective and granular way to measure what people remember from visualizations compared to previous methods.
  • Titles and themes are highly recallable: T-questions (about the title or theme) consistently showed the highest recall accuracy, suggesting that clear and informative titles are crucial for initial comprehension and retention.
  • Encoding duration matters, but not uniformly: While a longer encoding time (20 seconds vs. 10 seconds) generally improved recall, the effect was more pronounced for complex visualizations like tables and for questions requiring detailed data retrieval (F- and RV-questions).
  • Visualization complexity influences "I cannot remember" responses: Visualizations with higher complexity were more likely to elicit "I cannot remember" answers, suggesting a limit to how much detailed information can be processed and retained.
  • Recallability and Recognisability are linked but distinct: Visualizations that are memorable (easily recognized) are not necessarily easily recalled in detail. Highly memorable visualizations often had clear titles and explanatory text, while those with high recognisability but low recallability tended to be simpler, monotone plots.

Key Results

  • VisRecall Dataset: Contains 200 visualizations and 1,000 questions, with recallability scores from 305 participants.
  • Recall Accuracy: Participants could easily recognize most visualizations (average hit rate ~0.83) but only answered less than half of the recall questions correctly (average accuracy ~0.49).
  • Question Type Performance: T-questions had significantly higher recall accuracy (0.66) compared to other types (FE: 0.43, F: 0.55, RV: 0.46, U: 0.40). RV-questions had the highest rate of "I cannot remember" responses (24.7%).
  • Visualization Type Performance: Pie plots showed significantly higher recallability scores than other types.
  • RecallNet Performance: The proposed RecallNet model outperformed baseline methods (VGG-16, ResNet-34) in predicting both overall and fine-grained recallability scores, achieving a mean squared error (MSE) of 0.035 for overall recallability.

Limitations and Future Work

  • Dataset Size: The VisRecall dataset is relatively small due to the labor-intensive process of creating high-quality questions.
  • Qualitative Analysis: While the model shows predictive power, a deeper qualitative analysis is needed to link specific visualization features (like data-ink ratio or visual density) directly to recallability.
  • Future Work:
    • Expand VisRecall with more complex visualization types.
  • Investigate gaze behavior and visual attention during visualization interaction to better understand recall mechanisms.
  • Explore the application of recallability scores for visualization recommendation systems.
  • Develop more explainable models for recallability prediction.

Summary updated

Abstract

Despite its importance for assessing the effectiveness of communicating information visually, fine-grained recallability of information visualisations has not been studied quantitatively so far. In this work, we propose a question-answering paradigm to study visualisation recallability and present VisRecall — a novel dataset consisting of 200 visualisations that are annotated with crowd-sourced human (N = 305) recallability scores obtained from 1,000 questions of five question types. Furthermore, we present the first computational method to predict recallability of different visualisation elements, such as the title or specific data values. We report detailed analyses of our method on VisRecall and demonstrate that it outperforms several baselines in overall recallability and FE-, F-, RV-, and U-question recallability. Our work makes fundamental contributions towards a new generation of methods to assist designers in optimising visualisations.
BibTeX
@article{wang2022visrecall,
  author = {Yao Wang and Chuhan Jiao and Mihai Bâce and Andreas Bulling},
  title = {{VisRecall: Quantifying Information Visualisation Recallability via Question Answering}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2022},
  url = {https://openreview.net/forum?id=uCPwguVCHx},
}

Loading related content...

Figure