VProChart: Answering Chart Question Through Visual Perception Alignment Agent and Programmatic Solution Reasoning

Muye Huang, Lingling Zhang, Han Lai, Wenjun Wu, Xinyu Zhang, Jun Liu

Introduction

While Large Multimodal Models (LMMs) have advanced general image understanding, they frequently falter when interpreting charts. This difficulty stems from two primary challenges: perceptual complexity and reasoning complexity. Charts lack universal standards, requiring models to implicitly structure spatial relationships and semantic cues. Furthermore, multi-step logical or numerical operations often lead to "hallucinations" in purely neural architectures. VProChart addresses these hurdles by integrating a lightweight visual perception agent with a programmatic reasoning engine.

Methodology

The VProChart framework splits the Chart Question Answering (CQA) task into two specialized modules designed to handle perception and logic independently.

Visual Perception Alignment Agent (VPAgent)

The VPAgent treats charts not as flat pixel grids but as structured collections of elements. It employs three "Human Visual Perception" principles—Crosshair Intersecting, Spatial Proximity, and Color/Semantic Consistency—to align data points with their corresponding axes and labels. By using a Swin-Encoder for visual features and a Q-driven reasoning module, the agent focuses specifically on clusters relevant to the user's question.

The VProChart architecture, illustrating the Visual Perception Align Module and the Q-driven Chart Reasoning & QA Module.

Programmatic Solution Reasoning

To ensure mathematical and logical precision, VProChart moves away from "black-box" neural guessing. It utilizes a Large Language Model (LLM), such as Qwen2-7B-Instruct, to act as a programmer. The LLM parses natural language questions into Python-style programs that use a SUBSTEP() function to query the VPAgent for specific data points. A Python engine then executes these programs, ensuring that if the data extraction is accurate, the final calculation is perfect.

The programmatic reasoning flow, demonstrating how an LLM generates solution code for execution by a Python-style reasoning engine.

Evaluation and Key Results

VProChart was evaluated on three major benchmarks: ChartQA, PlotQA, and DVQA. The framework achieved state-of-the-art (SOTA) performance across these datasets. Notably, it reached 57.4% accuracy on the ChartQA-Human subset, nearly doubling the performance of earlier large-scale models like PaLI. These results suggest that explicit modeling of human perceptual biases is more effective for real-world charts than relying solely on synthetic pre-training.

Conclusion and Future Work

By separating perception from computation, VProChart avoids the "fuzzy math" problem common in standard LMMs. While the system currently depends on the LLM's ability to generate correct code and introduces some latency due to its two-stage process, it represents a significant step toward human-like chart understanding. Future research will explore expanding the framework to more complex schematic diagrams beyond standard bar and line charts.

Summary updated

Abstract

Charts are widely used for data visualization across various fields, including education, research, and business. Chart Question Answering (CQA) is an emerging task focused on the automatic interpretation and reasoning of data presented in charts. However, chart images are inherently difficult to interpret, and chart-related questions often involve complex logical and numerical reasoning, which hinders the performance of existing models. This paper introduces VProChart, a novel framework designed to address these challenges in CQA by integrating a lightweight Visual Perception Alignment Agent (VPAgent) and a Programmatic Solution Reasoning approach. VPAgent aligns and models chart elements based on principles of human visual perception, enhancing the understanding of chart context. The Programmatic Solution Reasoning approach leverages large language models (LLMs) to transform natural language reasoning questions into structured solution programs, facilitating precise numerical and logical reasoning. Extensive experiments on benchmark datasets such as ChartQA and PlotQA demonstrate that VProChart significantly outperforms existing methods, highlighting its capability in understanding and reasoning with charts.
BibTeX
@inproceedings{huang2025vprochart,
  author = {Muye Huang and Lingling Zhang and Han Lai and Wenjun Wu and Xinyu Zhang and Jun Liu},
  title = {{VProChart: Answering Chart Question Through Visual Perception Alignment Agent and Programmatic Solution Reasoning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i4.32384},
}
Figure