Introduction
While artificial intelligence has achieved significant milestones in describing natural scenes, the task of figure captioning—generating natural language descriptions for data visualizations like bar charts and line plots—remains a distinct challenge. Unlike natural images where objects are often spatially distinct, figures are dense with "pivot" elements such as labels, axes, and data points that require logical reasoning to interpret. This paper introduces a specialized framework designed to bridge this gap by enabling models to reason about data relations rather than just processing pixels.
The Challenge of Chart Interpretation
Standard image captioning models typically fail when applied to figures because they treat charts as collections of pixels rather than structured data representations. To generate accurate captions, a model must be able to:
- Identify specific labels and categories through text recognition.
- Understand comparative relationships (e.g., identifying the maximum value or comparing two bars).
- Maintain factual accuracy while producing grammatically natural, long-form descriptions.
Methodology: Feature, Label, and Relation Maps
The authors propose a novel architecture built on a ResNet-50 encoder and an LSTM decoder, enhanced by three specialized attention mechanisms:
- Feature Maps (Att_F): Standard spatial attention that focuses on the visual regions of the figure.
- Label Maps (Att_L): This module specifically attends to text labels extracted via OCR. By focusing on these "pivots," the model ensures it uses correct category names rather than hallucinating generic terms.
- Relation Maps (Att_R): The core innovation of this work. Inspired by Relation Networks, this module processes pairs of visual features through a Multi-Layer Perceptron (MLP) to learn abstract logical relations, such as "greater than" or "minimum value."
To further refine the output, the model is trained using Reinforcement Learning (RL) with a self-critical policy gradient. This approach rewards the model for generating full captions that score well on evaluation metrics like CIDEr, effectively mitigating the "exposure bias" common in standard cross-entropy training.
The FigCAP Dataset
To support this research, the authors introduced FigCAP, a large-scale synthetic dataset featuring five chart types (bar charts, pie charts, and various line plots). The dataset is split into two levels:
- FigCAP-H (High-level): Captions describing the figure type and general categories.
- FigCAP-D (Detailed): Complex captions describing specific data trends and comparative relations (e.g., "Category X is the maximum and is greater than Y").
Key Results and Performance
The proposed model, CapGen+Att_All+RL, demonstrated a dramatic improvement over baseline CNN-LSTM models on the FigCAP-D dataset:
- CIDEr Score: Increased from a baseline of 0.158 to 1.179.
- BLEU-4: Improved from 0.038 to 0.270.
Qualitatively, the model proved capable of generating sophisticated reasoning-based sentences, such as identifying specific colors as the maximum or minimum values and correctly ordering categories by magnitude. These results suggest that while general-purpose models struggle with figures, specialized attention to labels and pairwise relations makes automated chart reasoning feasible.
Limitations and Future Directions
Despite its success, the model occasionally suffers from "label confusion" (e.g., misidentifying similar shades of blue) or minor reasoning errors in complex rankings. Furthermore, because the FigCAP dataset is synthetic, future research will focus on applying these techniques to "in-the-wild" figures found in academic papers and financial reports, where visual noise and varied formatting present additional hurdles.
Summary updated