Figure Captioning with Reasoning and Sequence-Level Training

Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, Tong Yu, Ryan A. Rossi, Razvan C. BunescuarXiv 2019 · OpenReview

Abstract

Figures, such as bar charts, pie charts, and line plots, are widely used to convey important information in a concise format. They are usually human-friendly but difficult for computers to process automatically. In this work, we investigate the problem of figure captioning where the goal is to automatically generate a natural language description of the figure. While natural image captioning has been studied extensively, figure captioning has received relatively little attention and remains a challenging problem. First, we introduce a new dataset for figure captioning, FigCAP, based on FigureQA. Second, we propose two novel attention mechanisms. To achieve accurate generation of labels in figures, we propose Label Maps Attention. To model the relations between figure labels, we propose Relation Maps Attention. Third, we use sequence-level training with reinforcement learning in order to directly optimizes evaluation metrics, which alleviates the exposure bias issue and further improves the models in generating long captions. Extensive experiments show that the proposed method outperforms the baselines, thus demonstrating a significant potential for the automatic captioning of vast repositories of figures.

Introduction

While AI has become remarkably good at describing natural photos, "reading" a bar chart or line graph remains a significant challenge. Figures like pie charts and line plots are designed to be "human-friendly," condensing complex data into visual trends. However, for a computer, a figure is just a grid of pixels. Standard image captioning models struggle with figures due to logical complexity—requiring an understanding of relationships like "Company A has the highest revenue"—and sequence length, where long captions suffer from "exposure bias."

This paper introduces FigCAP, a new dataset for figure captioning, and a specialized model that uses "Relation Maps" and reinforcement learning to generate detailed, logical descriptions of data visualizations.

Methods

The authors propose an encoder-decoder architecture enhanced by three distinct attention mechanisms and a hybrid training strategy. A standard ResNet-50 extracts visual features, which are then processed by a Triple Attention Decoder.

Proposed architecture featuring Triple Attention mechanisms.

The decoder includes:

  • Feature Maps Attention ($Att\_F$): A standard spatial attention mechanism.
  • Label Maps Attention ($Att\_L$): A specialized layer focusing on specific text labels (categories) in the figure.
  • Relation Maps Attention ($Att\_R$): A novel component using a "Relation Network" to compare pairs of feature vectors, explicitly modeling logical relationships between graph elements.

To handle long captions, the authors employ Reinforcement Learning (Self-Critical Policy Gradient) to optimize the model directly for the CIDEr metric, alleviating the exposure bias inherent in traditional "Teacher Forcing."

Datasets

The authors introduced FigCAP, a large-scale synthetic dataset based on FigureQA. It consists of two versions:

  • FigCAP-H (High-level): Descriptions of figure types, number of labels, and category names.
  • FigCAP-D (Detailed): Complex logical descriptions, such as maximum/minimum values or trend intersections.

Both versions include approximately 100,000 training images across five figure types (bar charts, pie charts, line plots, etc.).

Results

The proposed model significantly outperformed the baseline CNN-LSTM across all metrics. As shown in the performance table for detailed captions, the inclusion of specialized attention and reinforcement learning leads to dramatic improvements.

Performance comparison on the FigCAP-D dataset.

Key insights from the results include:

  • Labels as Anchors: Attending specifically to labels ($Att\_L$) significantly improves accuracy, as these act as specific anchors in the figure.
  • Pairwise Reasoning: Explicitly calculating relations between feature vectors ($Att\_R$) allows the model to reason about relative properties (e.g., taller bars) without needing a massive unstructured memory.
  • RL for Long-Form Text: Reinforcement Learning proved essential for "Detailed" captions, preventing the model from losing coherence mid-sentence.

Conclusion

This work demonstrates that figure captioning can be effectively framed as a "self-asking VQA" task. While the current results on the synthetic FigCAP dataset are promising, future work must address the noise and stylistic variety of real-world figures. Potential directions include improving OCR dependency and exploring "inverting the visualization pipeline" to reconstruct raw data tables before caption generation.

Summary updated

Abstract

Figures, such as bar charts, pie charts, and line plots, are widely used to convey important information in a concise format. They are usually human-friendly but difficult for computers to process automatically. In this work, we investigate the problem of figure captioning where the goal is to automatically generate a natural language description of the figure. While natural image captioning has been studied extensively, figure captioning has received relatively little attention and remains a challenging problem. First, we introduce a new dataset for figure captioning, FigCAP, based on FigureQA. Second, we propose two novel attention mechanisms. To achieve accurate generation of labels in figures, we propose Label Maps Attention. To model the relations between figure labels, we propose Relation Maps Attention. Third, we use sequence-level training with reinforcement learning in order to directly optimizes evaluation metrics, which alleviates the exposure bias issue and further improves the models in generating long captions. Extensive experiments show that the proposed method outperforms the baselines, thus demonstrating a significant potential for the automatic captioning of vast repositories of figures.
BibTeX
@misc{chen2019figure,
  author = {Charles Chen and Ruiyi Zhang and Eunyee Koh and Sungchul Kim and Scott Cohen and Tong Yu and Ryan A. Rossi and Razvan C. Bunescu},
  title = {{Figure Captioning with Reasoning and Sequence-Level Training}},
  year = {2019},
  eprint = {1906.02850},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.02850},
}
Figure