VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning

Muye Huang, Lingling Zhang, Lai Han, Wenjun Wu, Xinyu Zhang, Jun Liu

TL;DR

VProChart is a new framework that masters Chart Question Answering (CQA) by combining a "visual perception" agent with a programmatic reasoning engine. By mimicking how humans group chart elements (like matching colors or aligning axes) and using an LLM to write Python-style code for complex math, it significantly outperforms existing models on benchmarks like ChartQA and PlotQA.

What Problem Does This Paper Address?

Charts are a unique challenge for AI. Unlike natural images, charts are abstract, structured data visualizations where meaning is derived from human-defined conventions—like legends, axes, and color-coding. Current models for Chart Question Answering (CQA) typically struggle with two things:

  1. Structural Interpretation: They often fail to grasp the relationship between a bar and its label, or a line and its entry in a legend, especially when layouts are non-standard.
  2. Complex Reasoning: Questions like "Is the sum of the two highest red bars greater than the blue bar in 2012?" require precise numerical calculation and multi-step logic that standard neural networks aren't naturally built to handle.

VProChart tackles these by explicitly modeling visual relationships and offloading the "math" to a structured programming approach.

Methods

The VProChart framework consists of two primary components:

1. Visual Perception Alignment Agent (VPAgent)

Instead of treating a chart as a flat grid of pixels, the VPAgent uses a Visual Perception Align Module to model relationships based on three human-like principles:

  • Crosshair Intersecting: Aligning elements horizontally and vertically (e.g., connecting a data point to its axis).
  • Spatial Proximity: Grouping elements that are physically close (e.g., a bar and its numerical label).
  • Color & Semantic Consistency: Linking elements that share the same color or meaning (e.g., matching a green bar to the green box in the legend).

This structure is illustrated in the system overview: Overview of the VPAgent architecture showing the encoding, clustering, and reasoning modules.

Figure 2: The overview of the proposed VPAgent. The proposed VPAgent comprises four modules: Chart & Question Encoder Module (left), Visual Perception Align Module (center), Q-driven Chart Reasoning and Question Answering Module (right).

2. Programmatic Solution Reasoning

For complex questions, VProChart doesn't just "guess" the answer. It uses a Large Language Model (LLM), specifically Qwen2-7B-Instruct, to act as a programmer. The LLM decomposes a natural language question into a Python-style solution program.

A reasoning engine then executes this code, calling the VPAgent as a tool to fetch specific values (using a SUBSTEP operator) and then performing the final calculations or comparisons programmatically.

Figure 3: An example of Programmatic Solution Reasoning.

Datasets

The authors evaluated VProChart on three major benchmarks:

  • ChartQA: Contains both human-curated (Human) and machine-generated (Augmented) charts. The "Human" subset is particularly difficult due to complex reasoning requirements.
  • PlotQA: A large-scale dataset (subsets D1 and D2) focused on scientific plots with heavy numerical reasoning.
  • DVQA: A synthetic dataset focused on chart-specific questions.

Key Insights and Ideas

  • Perception as a Prior: Explicitly modeling human visual perception principles (like color consistency) acts as a powerful inductive bias. The ablation studies showed that removing the "Visual Perception Alignment" module caused the largest drop in performance.
  • Decoupling Logic from Perception: By letting an LLM "plan" the math in code and the VPAgent "see" the data, the system avoids the common pitfalls where neural networks hallucinate arithmetic results.
  • Generalization to Real-World Charts: VProChart showed a massive improvement on the "Human" subset of ChartQA. This suggests that structured reasoning is more critical for messy, real-world charts than for clean, synthetic ones.

Key Results

VProChart achieved state-of-the-art (SOTA) results across the board:

  • ChartQA-Human: Reached 57.4% accuracy, outperforming the previous best model, ChartInstruct-P (50.1%), by over 7%.
  • PlotQA: On the D2 subset, it achieved 83.5%, a significant jump over many end-to-end visual models.
  • Efficiency: Even the standalone VPAgent (only 311M parameters) outperformed much larger models like UniChart on basic chart tasks, proving that smart architectural choices can beat raw scale.

Qualitative examples show VProChart correctly decomposing ratios and comparisons that trip up other SOTA models like MatCha or UniChart: Qualitative comparison showing VProChart's programmatic reasoning vs other models.

Figure 4: Two examples of VProChart. Above and below illustrate two examples from the PlotQA and ChartQA datasets, respectively.

Limitations and Future Work

While powerful, the framework relies on a two-stage process (LLM code generation followed by Agent execution), which can be computationally expensive. The authors also noted that while the programmatic approach handles math well, it still depends on the VPAgent's ability to correctly extract the initial values.

Future work will likely focus on tighter integration between human-like visual intuition and programmatic reasoning to handle even more ambiguous or stylistically unique diagrams.

Summary updated

Abstract

Charts are widely used for data visualization across various fields, including education, research, and business. Chart Question Answering (CQA) is an emerging task focused on the automatic interpretation and reasoning of data presented in charts. However, chart images are inherently difficult to interpret, and chart-related questions often involve complex logical and numerical reasoning, which hinders the performance of existing models. This paper introduces VProChart, a novel framework designed to address these challenges in CQA by integrating a lightweight Visual Perception Alignment Agent (VPAgent) and a Programmatic Solution Reasoning approach. VPAgent aligns and models chart elements based on principles of human visual perception, enhancing the understanding of chart context. The Programmatic Solution Reasoning approach leverages large language models (LLMs) to transform natural language reasoning questions into structured solution programs, facilitating precise numerical and logical reasoning. Extensive experiments on benchmark datasets such as ChartQA and PlotQA demonstrate that VProChart significantly outperforms existing methods, highlighting its capability in understanding and reasoning with charts.
BibTeX
@inproceedings{huang2025vprochart,
  author = {Muye Huang and Lingling Zhang and Lai Han and Wenjun Wu and Xinyu Zhang and Jun Liu},
  title = {{VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i4.32384},
}

Loading related content...

Figure