GoT-CQA: Graph-of-Thought Guided Compositional Reasoning for Chart Question Answering

Lingling Zhang, Muye Huang, QianYing Wang, Yaxian Wang, Wenjun Wu, Jun LiuarXiv 2024 · OpenReview

Abstract

Chart Question Answering (CQA) aims at answering questions based on visual chart content, playing a vital role in summarization, business analysis, and report generation. CQA is a challenging multi-modal task due to strong context dependence and complex reasoning requirements. Context dependence requires answers to be strictly derived from visual or internal data, while reasoning involves various logical and numerical operations. In this paper, we propose a novel Graph-of-Thought (GoT) guided compositional reasoning model, GoT-CQA. We transform chart-oriented questions into a directed acyclic GoT composed of multiple operator nodes (localization, numerical, and logical). Guided by this GoT, our framework executes multi-step reasoning operations. Experiments on ChartQA and PlotQA-D datasets demonstrate that GoT-CQA achieves outstanding performance, particularly on complex human-written reasoning questions.

The Challenge of Chart Question Answering

Chart Question Answering (CQA) is significantly more difficult than standard Visual Question Answering (VQA). While VQA often relies on "common sense" or statistical priors, CQA requires strong context dependence. Answers cannot be guessed; they must be extracted through precise analysis of specific data points within the chart.

Current research faces two primary hurdles:

  1. Multi-step Reasoning: Questions like "Are the number of descendants of P2 more than the average?" require a model to locate specific bars, retrieve values, calculate an average, and perform a comparison.
  2. Unified Frameworks: Most models struggle to generalize across different question types—such as structural understanding versus complex data retrieval—without becoming overly specialized or computationally expensive.

The GoT-CQA Framework

The core innovation of this work is the Graph-of-Thought (GoT) guided compositional reasoning. The system consists of three main modules:

  • Chart & Question Parsing: The model utilizes Donut, an OCR-free transformer architecture, to encode the chart into a visual feature sequence. Simultaneously, the question is transformed into a Directed Acyclic Graph (DAG) where nodes represent specific operators.
  • Compositional Reasoning Module: Guided by the GoT, this module triggers specific reasoning blocks in sequence: Localization (Loc) for position information, Numerical (Num) for value retrieval/calculation, and Logical (Log) for higher-level operations like comparison or summation.
  • Answering Module: A transformer decoder (based on mBART) takes the final data flow from the reasoning module to generate a natural language or numerical answer.
Overview of the GoT-CQA framework illustrating the pipeline from chart/question parsing to answer prediction.

Unlike "black-box" models, GoT-CQA is auto-compositional, meaning the architecture of the reasoning network dynamically adapts based on the specific logic required by the question.

Graph-of-Thought and Compositional Reasoning

Most large vision-language models (LVLMs) attempt to reason implicitly within their hidden layers. GoT-CQA argues that making reasoning steps explicit (Loc → Num → Log) leads to better accuracy and interpretability. By decomposing a question into a graph of operators, the model can focus on one sub-task at a time, reducing the cognitive load on the neural network.

Examples of GoT structures for various question types and the internal attention-based reasoning blocks.

Experimental Results

The authors evaluated GoT-CQA on two major benchmarks: ChartQA and PlotQA-D.

  • ChartQA Superiority: GoT-CQA achieved an overall score of 67.5%, notably reaching 47.1% on human-written questions. This represents a significant improvement over previous state-of-the-art models like UniChart (43.9%) and MatCha (38.2%). The model shows its greatest strength on human-authored questions, which are often more linguistically diverse and logically complex than template-generated ones.
  • PlotQA Performance: On the PlotQA-D1 reasoning subset, GoT-CQA reached 86.5%, outperforming the previous best (CRCT) by over 30 percentage points.
  • Ablation Success: Removing the GoT guidance (treating the question as a simple sentence embedding) caused performance to drop across all metrics, confirming that the graph structure is essential for complex reasoning.

Conclusion

GoT-CQA demonstrates that explicit reasoning chains significantly enhance a model's ability to handle complex, human-authored chart questions. While the reasoning logic is robust, performance is still occasionally limited by the initial visual feature extraction. Future work will explore more diverse operators and potentially integrate more powerful visual encoders to improve initial data extraction in cluttered or complex chart types.

Summary updated

Abstract

Chart Question Answering (CQA) aims at answering questions based on the visual chart content, which plays an important role in chart sumarization, business data analysis, and data report generation. CQA is a challenging multi-modal task because of the strong context dependence and complex reasoning requirement. The former refers to answering this question strictly based on the analysis of the visual content or internal data of the given chart, while the latter emphasizes the various logical and numerical reasoning involved in answer prediction process. In this paper, we pay more attention on the complex reasoning in CQA task, and propose a novel Graph-of-Thought (GoT) guided compositional reasoning model called GoT-CQA to overcome this problem. At first, we transform the chart-oriented question into a directed acyclic GoT composed of multiple operator nodes, including localization, numerical and logical operator. It intuitively reflects the human brain's solution process to this question. After that, we design an efficient auto-compositional reasoning framework guided by the GoT, to excute the multi-step reasoning operations in various types of questions. Comprehensive experiments on ChartQA and PlotQA-D datasets show that GoT-CQA achieves outstanding performance, especially in complex human-written and reasoning questions, comparing with the latest popular baselines.
BibTeX
@misc{zhang2024got,
  author = {Lingling Zhang and Muye Huang and QianYing Wang and Yaxian Wang and Wenjun Wu and Jun Liu},
  title = {{GoT-CQA: Graph-of-Thought Guided Compositional Reasoning for Chart Question Answering}},
  year = {2024},
  eprint = {2409.02611},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.02611},
}
Figure