LineCap: Line Charts for Data Visualization Captioning Models

Anita Mahinpei, Zona Kostic, Chris Tanner

Abstract

Data visualization captions help readers understand the purpose of a visualization and are crucial for individuals with visual impairments. The prevalence of poor figure captions and the successful application of deep learning approaches to image captioning motivate the use of similar techniques for automated figure captioning. However, research in this field has been stunted by the lack of suitable datasets. We introduce LineCap, a novel figure captioning dataset of 3,528 figures, and we provide insights from curating this dataset and using end-to-end deep learning models for automated figure captioning.

Introduction

For researchers with visual impairments, a data visualization without a descriptive caption represents a significant barrier to information. While general image captioning—such as identifying objects in natural scenes—has advanced rapidly, automated figure captioning for scientific charts has lagged behind. This stagnation is primarily due to two bottlenecks: the lack of high-quality datasets and the inherent complexity of scientific plots, which often feature overlapping lines and intricate LaTeX symbols.

Existing datasets frequently rely on synthetic charts that are too simplistic or web-scraped captions that provide either trivial information (e.g., "Figure 1") or highly contextual "Level 4" insights that require reading the entire paper to comprehend. LineCap aims to bridge this gap by providing a benchmark for "Level 2 and 3" captions—descriptions that focus on descriptive statistics, extrema, and complex trends (e.g., "Line 1 increases rapidly then plateaus").

The LineCap Dataset

LineCap consists of 3,528 real-world line charts sampled from the SciCap dataset, which contains figures from arXiv papers. The dataset includes 3,964 human-authored annotations. To handle the complexity of scientific notation, the researchers implemented standardized terminology. Human annotators were instructed to use tokens like xlabel, ylabel, and Line 1 through Line 5. This approach allows models to learn the underlying structure of a trend without being distracted by complex variable names or LaTeX formatting.

Methodology

The researchers proposed a two-stage deep learning pipeline to tackle the captioning task:

  1. Line Count Prediction: Before generating a description, the model must determine the number of data series present. A DenseNet architecture combined with "fusion blocks" was used to extract both high-level and low-level visual features to predict the line count (ranging from 1 to 5).
  2. Iterative Caption Generation: Rather than describing an entire multi-line chart in one pass, the model iterates through each line. For a three-line chart, the model runs three times, taking the image, the total line count, and a specific line index (e.g., "Describe Line 2") as input.

Key Results and Insights

The experimental results revealed both the potential and the current limitations of automated chart captioning:

  • Line Counting Accuracy: The model was highly effective at identifying the number of lines in a chart, achieving 94.05% accuracy on the test set.
  • Captioning Performance: On a human-rated scale of 1 (completely inaccurate) to 5 (completely accurate), the model averaged 3.20. Performance was strongest on single-line charts (3.43) but declined as complexity increased.
  • The "Identity Crisis": In multi-line charts, the model frequently struggled with visual grounding. It often suffered from an "identity crisis," where it would repeat the same description for every line in the chart, unable to distinguish which visual features belonged to which specific index, especially when lines crossed or overlapped.
  • Metric Mismatch: A significant finding was that standard NLP metrics like BLEU-4 and CIDEr correlate poorly with human judgments of chart accuracy. While BLEURT performed slightly better, the study highlights a dire need for automated metrics specifically designed for data visualization, where the direction and order of a trend are more critical than vocabulary overlap.

Conclusion and Future Work

LineCap represents a major step toward making scientific data more accessible. By focusing on trend-based descriptions rather than simple label reading, the dataset challenges models to "see" the data more deeply. Future research must focus on improving spatial grounding to resolve the identity crisis in multi-line plots and expanding the dataset to include other visualization types such as bar charts and scatter plots.

Summary updated

Abstract

Data visualization captions help readers understand the purpose of a visualization and are crucial for individuals with visual impairments. The prevalence of poor figure captions and the successful application of deep learning approaches to image captioning motivate the use of similar techniques for automated figure captioning. However, research in this field has been stunted by the lack of suitable datasets. We introduce LineCap, a novel figure captioning dataset of 3,528 figures, and we provide insights from curating this dataset and using end-to-end deep learning models for automated figure captioning.
BibTeX
@inproceedings{mahinpei2022linecap,
  author = {Anita Mahinpei and Zona Kostic and Chris Tanner},
  title = {{LineCap: Line Charts for Data Visualization Captioning Models}},
  booktitle = {IEEE VIS (Short Papers)},
  year = {2022},
  url = {https://openreview.net/forum?id=q8kR6E1uvh},
}
Figure