StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding

Renqiu Xia, Bo Zhang, Haoyang Peng, Ning Liao, Peng Ye, Botian Shi, Junchi Yan, Yu QiaoSubmission to ICLR 2024 · Rejected

TL;DR

StructChart introduces a label-efficient, two-stage paradigm for visual chart understanding that bridges the gap between low-level visual perception and high-level reasoning. By replacing fragile linearized CSV strings with robust Structured Triplet Representations (STR) and augmenting training with an LLM-synthesized dataset (SimChart9K), StructChart achieves state-of-the-art visual reasoning while matching full-data performance using only 20% of real-world annotations.

What Problem Does This Paper Address?

Extracting data from charts and reasoning over it (e.g., answering questions, writing summaries, or redrawing visual plots) remains difficult for multi-modal models due to three key bottlenecks:

  1. The Perception-Reasoning Gap: Traditional vision models extract raw text or tables, while language models must perform complex numerical comparisons across rows and columns. Existing pipelines commonly feed linearized CSV tokens (LCT) into language models, but LCT is fragile—an extra separator comma or minor row reordering easily breaks downstream parsing.
  2. Incomplete Perception Metrics: Existing chart extraction benchmarks focus on specific chart types (e.g., simple bar plots) and lack metrics that account for row/column permutation invariance or mixed numerical/textual errors.
  3. Expensive Annotation: Collecting and manually annotating diverse real-world charts across scientific, financial, and medical domains is prohibitively labor-intensive.

Methods

StructChart tackles visual chart understanding via a modular two-stage architecture:

[ Chart Image ] 
       │
       ▼  (Chart-oriented Information Extractor - ViT Encoder-Decoder)
[ Linearized CSV Tokens (LCT) ]
       │
       ▼  (Triplet Structuring)
[ Structured Triplet Representations (STR): (Row_Entity, Col_Entity, Value) ]
       │
       ▼  (Downstream Reasoning via LLM: GPT-3.5)
[ QA / Summarization / Chart Redrawing ]
  • Chart-oriented Information Extractor (CIE): A Vision Transformer (ViT) encoder-decoder that dynamically scales input images into a fixed number of patches fitting the maximum sequence length, paired with 2D absolute positional embeddings to handle arbitrary resolutions.
  • Structured Triplet Representations (STR): Instead of flattened CSVs, chart data is reformulated into relational triplets: $$\left(\text{Entity}_{\text{row}}, \text{Entity}_{\text{column}}, \text{Value}\right)$$ This format provides permutation invariance over row/column reorderings and protects against parsing errors caused by text strings containing commas.

  • Structuring Chart-oriented Representation Metric (SCRM): A unified metric that compares predicted and ground-truth triplets by measuring normalized string edit distance for entities and relative numerical error for values across three tolerance regimes (strict, slight, and high), aggregating into mean Precision ($m\text{Precision}$) across IoU thresholds from $0.50$ to $0.95$.

  • LLM-Based Self-Inspection Data Production: The authors prompt GPT-3.5 in a few-shot setting to generate diverse statistical CSV tables, followed by automated Python/matplotlib script generation. An automated self-inspection loop executes the rendering code and automatically discards non-executable scripts.

Datasets

The authors evaluate on multiple established benchmarks and introduce a new synthetic dataset:

  • SimChart9K (Introduced): 9,098 synthetic chart images paired with clean CSV annotations generated using the LLM-driven self-inspection pipeline.
  • ChartQA: 20,882 real-world charts with visual/logical question-answering pairs.
  • PlotQA: Synthetic plot dataset containing over 224,000 plots and 28.9 million QA pairs.
  • FigureQA: Rule-based synthetic visual reasoning dataset with 15 question templates.
  • Chart2Text: 8,305 real-world charts scraped from Statista for chart summarization.

Key Insights and Ideas

  • Structure Trumps Raw Text Formatting: Standard linearized CSV strings break when column values contain commas or when rows are transposed. Structuring chart information into explicit relational triplets (Row, Column, Value) dramatically improves LLM comprehension without requiring complex prompt engineering.
  • LLM-Driven Chart Synthesis Closes the Data Gap: Because statistical tables and matplotlib scripts can be generated cheaply by LLMs, synthetic data distributions can closely match real-world feature spaces (verified via t-SNE embeddings).
  • High Label Efficiency: Pre-training on synthetic charts drastically cuts down the requirement for real annotated charts—allowing small datasets to reach parity with models trained on full corpora.

Key Results

  • Downstream Question Answering: On the ChartQA benchmark, StructChart + GPT-3.5 achieves 65.3% average accuracy (83.9% on the augmented split, 46.7% on the human split), outperforming:
  • DePlot + GPT-3.5 under identical prompting settings (52.9%, a +12.4% gain),
  • MatCha (64.2%), despite MatCha pre-training on closed-source web chart data.
  • Moving from flat LCT to triplet STR representation alone boosted QA accuracy from 56.3% to 65.3%.
  • Label Efficiency:
  • On ChartQA, training with only 20% real data + SimChart9K achieves $0.6465$ strict $m\text{Precision}$ (approaching the $0.6770$ achieved with 100% real data).
  • With 50% real data + SimChart9K, the model scores 0.6902 strict $m\text{Precision}$, strictly outperforming models trained on the entire real dataset without synthetic augmentation ($0.6770$).
  • On PlotQA, using just 10% real data + SimChart9K matches full-dataset performance.
  • FigureQA Valuation: StructChart reaches 63.3% validation accuracy, beating DePlot + GPT-3.5 (53.9%).

Limitations and Future Work

  • Two-Stage Disconnection: Perception and reasoning remain split into two independent modules (ViT extractor followed by frozen LLM reasoning). The authors identify an end-to-end differentiable architecture incorporating STR extraction as an important open challenge.
  • Perception Hallucinations on Highly Cluttered Backgrounds: While StructChart outperforms prior work when processing noisy document backgrounds, extreme visual clutter or non-standard visual glyphs can still yield extraction errors.
  • Ethical Considerations: Automated high-fidelity chart redrawing creates potential risks around chart plagiarism or academic misconduct, though the authors note that STR extraction can conversely be used for duplicate and plagiarism detection.

Summary updated

Abstract

Charts are common in literature across different scientific fields, conveying rich information easily accessible to readers. Current chart-related tasks focus on either chart perception which refers to extracting information from the visual charts, or performing reasoning given the extracted data, e.g. in a tabular form. In this paper, we aim to establish a unified and label-efficient learning paradigm for joint perception and reasoning tasks, which can be generally applicable to different downstream tasks, beyond the question-answering task as specifically studied in peer works. Specifically, StructChart first reformulates the chart information from the popular tubular form (specifically linearized CSV) to the proposed Structured Triplet Representations (STR), which is more friendly for reducing the task gap between chart perception and reasoning due to the employed structured information extraction for charts. We then propose a Structuring Chart-oriented Representation Metric (SCRM) to quantitatively evaluate the performance for the chart perception task. To enrich the dataset for training, we further explore the possibility of leveraging the Large Language Model (LLM), enhancing the chart diversity in terms of both chart visual style and its statistical information. Extensive experiments are conducted on various chart-related tasks, demonstrating the effectiveness and promising potential for a unified chart perception-reasoning paradigm to push the frontier of chart understanding.
BibTeX
@misc{xia2023structchart,
  author = {Renqiu Xia and Bo Zhang and Haoyang Peng and Ning Liao and Peng Ye and Botian Shi and Junchi Yan and Yu Qiao},
  title = {{StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding}},
  year = {2023},
  eprint = {2309.11268},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.11268},
}

Loading related content...

Figure