Figure Captioning with Relation Maps for Reasoning

Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, Ryan A. Rossi

Introduction

While artificial intelligence has achieved significant milestones in describing natural scenes, the task of figure captioning—generating natural language descriptions for data visualizations like bar charts and line plots—remains a distinct challenge. Unlike natural images where objects are often spatially distinct, figures are dense with "pivot" elements such as labels, axes, and data points that require logical reasoning to interpret. This paper introduces a specialized framework designed to bridge this gap by enabling models to reason about data relations rather than just processing pixels.

The Challenge of Chart Interpretation

Standard image captioning models typically fail when applied to figures because they treat charts as collections of pixels rather than structured data representations. To generate accurate captions, a model must be able to:

  1. Identify specific labels and categories through text recognition.
  2. Understand comparative relationships (e.g., identifying the maximum value or comparing two bars).
  3. Maintain factual accuracy while producing grammatically natural, long-form descriptions.
An example horizontal bar chart from the FigCAP dataset, illustrating the 'pivot' elements like labels and data bars that the model must interpret.

Methodology: Feature, Label, and Relation Maps

The authors propose a novel architecture built on a ResNet-50 encoder and an LSTM decoder, enhanced by three specialized attention mechanisms:

  • Feature Maps (Att_F): Standard spatial attention that focuses on the visual regions of the figure.
  • Label Maps (Att_L): This module specifically attends to text labels extracted via OCR. By focusing on these "pivots," the model ensures it uses correct category names rather than hallucinating generic terms.
  • Relation Maps (Att_R): The core innovation of this work. Inspired by Relation Networks, this module processes pairs of visual features through a Multi-Layer Perceptron (MLP) to learn abstract logical relations, such as "greater than" or "minimum value."
The proposed model architecture featuring the integration of Feature, Label, and Relation Map attention modules feeding into an LSTM decoder.

To further refine the output, the model is trained using Reinforcement Learning (RL) with a self-critical policy gradient. This approach rewards the model for generating full captions that score well on evaluation metrics like CIDEr, effectively mitigating the "exposure bias" common in standard cross-entropy training.

The FigCAP Dataset

To support this research, the authors introduced FigCAP, a large-scale synthetic dataset featuring five chart types (bar charts, pie charts, and various line plots). The dataset is split into two levels:

  • FigCAP-H (High-level): Captions describing the figure type and general categories.
  • FigCAP-D (Detailed): Complex captions describing specific data trends and comparative relations (e.g., "Category X is the maximum and is greater than Y").

Key Results and Performance

The proposed model, CapGen+Att_All+RL, demonstrated a dramatic improvement over baseline CNN-LSTM models on the FigCAP-D dataset:

  • CIDEr Score: Increased from a baseline of 0.158 to 1.179.
  • BLEU-4: Improved from 0.038 to 0.270.

Qualitatively, the model proved capable of generating sophisticated reasoning-based sentences, such as identifying specific colors as the maximum or minimum values and correctly ordering categories by magnitude. These results suggest that while general-purpose models struggle with figures, specialized attention to labels and pairwise relations makes automated chart reasoning feasible.

Limitations and Future Directions

Despite its success, the model occasionally suffers from "label confusion" (e.g., misidentifying similar shades of blue) or minor reasoning errors in complex rankings. Furthermore, because the FigCAP dataset is synthetic, future research will focus on applying these techniques to "in-the-wild" figures found in academic papers and financial reports, where visual noise and varied formatting present additional hurdles.

Summary updated

Abstract

Figures, such as line plots, pie charts, bar charts, are widely used to convey important information in a concise format. In this work, we investigate the problem of figure caption generation where the goal is to automatically generate a natural language description for a given figure. While natural image captioning has been studied extensively, figure captioning has received relatively little attention and remains a challenging problem. A successful solution to this task has many potential applications, such as: 1) automatic parsing large amount of figures in PDF document; 2) improving user experience by allowing figure content to be accessible to those with visual impairment. To solve this problem, we introduce a dataset FigCAP and propose novel attention mechanism. In order to solve the exposure bias issue, we further train the captioning model with sequence-level policy based on reinforcement learning, which directly optimizes evaluation metrics. Extensive experiments show that the proposed method outperforms the baselines, thus demonstrating a significant potential for automatic generating captions for figures.
BibTeX
@inproceedings{chen2020figure,
  author = {Charles Chen and Ruiyi Zhang and Eunyee Koh and Sungchul Kim and Scott Cohen and Ryan A. Rossi},
  title = {{Figure Captioning with Relation Maps for Reasoning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093592},
}
Figure