LineCap: Line Charts for Data Visualization Captioning Models

Anita Mahinpei, Zona Kostic, Chris Tanner

TL;DR

LineCap is a new dataset of over 3,500 real-world line charts from scientific papers, paired with human-authored captions that describe high-level trends rather than just reading labels. The researchers developed a two-stage deep learning pipeline to count lines and generate descriptions, revealing that while AI is getting better at "seeing" simple charts, multi-line scientific plots remain a significant hurdle for current models.

What Problem Does This Paper Address?

For researchers with visual impairments, a data visualization without a descriptive caption is a brick wall. While we have made massive strides in general image captioning (e.g., "a dog catching a frisbee"), automated figure captioning for scientific charts has lagged behind.

The authors identify two main bottlenecks:

  1. Data Quality: Existing datasets often rely on synthetic charts (which are too simple) or web-scraped captions that are either too brief ("Figure 1") or contain "Level 4" information—contextual insights that require reading the entire paper to understand.
  2. Complexity: Scientific charts use complex LaTeX symbols, subscripts, and overlapping lines that break standard OCR and captioning models.

LineCap aims to bridge this gap by providing a benchmark for "Level 2 and 3" captions—descriptions that focus on descriptive statistics, extrema, and complex trends (e.g., "Line 1 increases rapidly then plateaus").

Methods

The authors propose a two-stage deep learning pipeline:

  • Stage 1: Line Count Prediction. Before describing a chart, the model must know how many lines it is looking at. They used a DenseNet architecture combined with "fusion blocks" to extract both high-level and low-level visual features to predict the number of lines (from 1 to 5).
  • Stage 2: Iterative Caption Generation. Instead of describing the whole chart at once, the model iterates. For a 3-line chart, it runs three times. Each pass takes the image, the total line count, and the specific line index (e.g., "Describe Line 2") as input.
  • Standardized Terminology: To avoid the "math symbol" problem, they instructed human annotators to use tokens like xlabel, ylabel, and Line 1 through Line 5. This allows the model to learn the structure of the trend without getting tripped up by complex variable names.

Datasets

  • LineCap (New): The star of the paper. It contains 3,528 line charts sampled from the SciCap dataset (which itself contains figures from arXiv papers). It includes 3,964 human-authored annotations.
  • FigureQA: A synthetic dataset used to pre-train the line-counting model.
  • SciCap: Used as the source for the raw scientific images.

Key Insights and Ideas

  • The "Level" Matters: The authors lean on the framework that captions should move beyond "Level 1" (just reading the axes) to "Level 2/3" (trends and comparisons). This is what users with visual impairments actually find most useful.
  • Standardization over Raw OCR: By replacing complex axis labels with standardized tokens, the researchers created a "plug-and-play" system. A separate chart-parsing model could extract the actual text labels, which are then swapped back into the generated caption.
  • Metric Mismatch: One of the most important findings is that standard NLP metrics like BLEU-4 and CIDEr (which look for word overlaps) correlate very poorly with human judgments of chart accuracy. BLEURT, a metric based on pre-trained language models, performed slightly better but is still not a perfect proxy for "truth" in a chart.

Key Results

  • Line Counting: The model was highly effective at counting lines, achieving 94.05% accuracy on the test set.
  • Captioning Accuracy: On a human-rated scale of 1 (completely inaccurate) to 5 (completely accurate), the model averaged 3.20.
  • Single vs. Multi-line: The model performed best on single-line charts (3.43 accuracy). However, for multi-line charts, it often struggled with "identity crisis"—it would frequently repeat the same description for every line in the chart, unable to distinguish which visual features belonged to which index.

Limitations and Future Work

  • Visual Grounding: The current model lacks a strong "spatial" understanding. It doesn't always know which line is "Line 1" versus "Line 2" if they cross or overlap significantly.
  • Dataset Diversity: While LineCap is a major step forward, it is limited to line charts with up to five lines. Future work needs to expand this to bar charts, scatter plots, and more complex multi-panel figures.
  • The Need for Better Metrics: The authors call for the development of new automated metrics specifically designed for data visualization, where the order and direction of a trend are more important than just using the right vocabulary.

Summary updated

Abstract

Data visualization captions help readers understand the purpose of a visualization and are crucial for individuals with visual impairments. The prevalence of poor figure captions and the successful application of deep learning approaches to image captioning motivate the use of similar techniques for automated figure captioning. However, research in this field has been stunted by the lack of suitable datasets. We introduce LineCap, a novel figure captioning dataset of 3,528 figures, and we provide insights from curating this dataset and using end-to-end deep learning models for automated figure captioning.
BibTeX
@inproceedings{mahinpei2022linecap,
  author = {Anita Mahinpei and Zona Kostic and Chris Tanner},
  title = {{LineCap: Line Charts for Data Visualization Captioning Models}},
  booktitle = {IEEE VIS (Short Papers)},
  year = {2022},
  url = {https://openreview.net/forum?id=q8kR6E1uvh},
}

Loading related content...

Figure