StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding

Renqiu Xia, Bo Zhang, Haoyang Peng, Ning Liao, Peng Ye, Botian Shi, Junchi Yan, Yu Qiao

Abstract

Charts are common in literature across different scientific fields, conveying rich information easily accessible to readers. Current chart-related tasks focus on either chart perception which refers to extracting information from the visual charts, or performing reasoning given the extracted data, e.g. in a tabular form. In this paper, we aim to establish a unified and label-efficient learning paradigm for joint perception and reasoning tasks, which can be generally applicable to different downstream tasks, beyond the question-answering task as specifically studied in peer works. Specifically, StructChart first reformulates the chart information from the popular tubular form (specifically linearized CSV) to the proposed Structured Triplet Representations (STR), which is more friendly for reducing the task gap between chart perception and reasoning due to the employed structured information extraction for charts. We then propose a Structuring Chart-oriented Representation Metric (SCRM) to quantitatively evaluate the performance for the chart perception task. To enrich the dataset for training, we further explore the possibility of leveraging the Large Language Model (LLM), enhancing the chart diversity in terms of both chart visual style and its statistical information. Extensive experiments are conducted on various chart-related tasks, demonstrating the effectiveness and promising potential for a unified chart perception-reasoning paradigm to push the frontier of chart understanding.

Introduction

While Large Multimodal Models (LMMs) have made significant strides in general image understanding, charts present a unique challenge. They are not merely pictures but visual encodings of structured statistical data. Current approaches often suffer from a "Perception-Reasoning Gap," where models extract data as flat strings (e.g., linearized CSV), losing the subtle structural relationships between headers and values necessary for complex logical reasoning. Additionally, high-quality annotated chart data is expensive to produce, and existing metrics for chart perception are often too simplistic to account for underlying data structures.

Methodology

The StructChart framework introduces a two-stage pipeline designed to bridge the gap between seeing a chart and reasoning about its data. The system consists of four key components:

  • Chart-oriented Information Extractor (CIE): A vision-transformer (ViT) encoder and a text decoder that transform a chart image into a text-level representation.
  • Structured Triplet Representation (STR): Instead of a standard flat CSV string, the model outputs data as triplets: $(Entity_{row}, Entity_{column}, Value)$. This format is robust to structural variations and explicitly links data points to their headers.
  • Structuring Chart-oriented Representation Metric (SCRM): A new evaluation metric designed specifically for STR, using edit distance for text labels and relative error for numerical values.
  • PlotAgent: An LLM-based data augmentation engine that uses "self-inspection" to generate diverse statistical data and corresponding Python code to render new, stylistically varied charts.
The StructChart pipeline: from LLM-based data production and CIE training to representation transformation and downstream task execution.

Datasets and Data Augmentation

The study utilizes several benchmark datasets, including ChartQA, PlotQA, and Chart2Text. To boost performance in low-data scenarios, the authors introduced SimChart9K, a dataset of 9,536 charts generated via PlotAgent. Experiments demonstrated that synthetic data is a powerful tool; a model trained on only 20% of real-world data supplemented by SimChart9K could match the performance of a model trained on 100% real data.

Key Insights and Results

The primary innovation of StructChart is the shift from sequence-based to structure-based representation. By forcing the model to output triplets (STR), the researchers found that LLMs could reason much more effectively because the relationship between a number and its context is never lost. This is particularly evident in complex scenarios like 3D charts.

Example of Structured Triplet Representation (STR) for a 3D chart, showing how complex visual data is mapped into clear (Entity, Entity, Value) triplets.

StructChart achieved state-of-the-art results, with a combined GPT-4 approach reaching an average score of 78.5% on the ChartQA test set, surpassing the end-to-end GPT-4V model (75.5%). Furthermore, the two-stage pipeline (Image $\rightarrow$ STR $\rightarrow$ Reasoning) provides high interpretability, allowing for clear error analysis between the perception and reasoning stages.

Conclusion

StructChart establishes a unified and label-efficient learning paradigm for chart understanding. By leveraging structured representations and LLM-driven data augmentation, it pushes the frontier of visual reasoning. Future work will focus on integrating external multimodal context and exploring statistical patterns for time-series forecasting.

Summary updated

Abstract

Charts are common in literature across different scientific fields, conveying rich information easily accessible to readers. Current chart-related tasks focus on either chart perception which refers to extracting information from the visual charts, or performing reasoning given the extracted data, e.g. in a tabular form. In this paper, we aim to establish a unified and label-efficient learning paradigm for joint perception and reasoning tasks, which can be generally applicable to different downstream tasks, beyond the question-answering task as specifically studied in peer works. Specifically, StructChart first reformulates the chart information from the popular tubular form (specifically linearized CSV) to the proposed Structured Triplet Representations (STR), which is more friendly for reducing the task gap between chart perception and reasoning due to the employed structured information extraction for charts. We then propose a Structuring Chart-oriented Representation Metric (SCRM) to quantitatively evaluate the performance for the chart perception task. To enrich the dataset for training, we further explore the possibility of leveraging the Large Language Model (LLM), enhancing the chart diversity in terms of both chart visual style and its statistical information. Extensive experiments are conducted on various chart-related tasks, demonstrating the effectiveness and promising potential for a unified chart perception-reasoning paradigm to push the frontier of chart understanding.
BibTeX
@misc{xia2023structchart,
  author = {Renqiu Xia and Bo Zhang and Haoyang Peng and Ning Liao and Peng Ye and Botian Shi and Junchi Yan and Yu Qiao},
  title = {{StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding}},
  year = {2023},
  eprint = {2309.11268},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.11268},
}
Figure