Abstract
Charts are widely used for data visualization across various fields, including education, research, and business. Chart Question Answering (CQA) is an emerging task focused on the automatic interpretation and reasoning of data presented in charts. However, chart images are inherently difficult to interpret, and chart-related questions often involve complex logical and numerical reasoning, which hinders the performance of existing models. This paper introduces VProChart, a novel framework designed to address these challenges in CQA by integrating a lightweight Visual Perception Alignment Agent (VPAgent) and a Programmatic Solution Reasoning approach. VPAgent aligns and models chart elements based on principles of human visual perception, enhancing the understanding of chart context. The Programmatic Solution Reasoning approach leverages large language models (LLMs) to transform natural language reasoning questions into structured solution programs, facilitating precise numerical and logical reasoning. Extensive experiments on benchmark datasets such as ChartQA and PlotQA demonstrate that VProChart significantly outperforms existing methods, highlighting its capability in understanding and reasoning with charts.
Introduction
Charts are intentional data visualizations that rely on abstract conventions like legends, grid lines, and color-coding. Unlike natural images, interpreting charts requires specific "visual alignment" to link data points to their labels. Furthermore, Chart Question Answering (CQA) often demands multi-step logic and precise arithmetic (e.g., comparing sums across different years). Existing end-to-end models frequently treat charts as black boxes, leading to failures in linking data points or performing complex calculations.
VProChart addresses these challenges by combining a lightweight Visual Perception Alignment Agent (VPAgent) with a Programmatic Solution Reasoning approach. This framework mimics human perception of chart structures and uses Large Language Models (LLMs) to translate complex queries into executable code, ensuring both structural understanding and numerical precision.
Methodology
The VProChart architecture consists of two primary components designed to bridge the gap between visual perception and logical reasoning.
Visual Perception Alignment Agent (VPAgent)
The VPAgent (311M parameters) is designed to model chart structures by mimicking human visual perception. It employs three core principles:
- Crosshair Intersecting: Aligning data points (such as bars or line vertices) with their corresponding X and Y axes.
- Spatial Proximity: Linking labels to the chart elements they are physically closest to.
- Color & Semantic Consistency: Matching legend items to chart elements based on color and text meaning.
Programmatic Solution Reasoning
To handle complex logic, VProChart uses a Large Language Model (specifically Qwen2-7B-Instruct) to act as a "programmer." Instead of directly predicting an answer, the LLM generates a Python-style program that decomposes the question into sub-steps. These programs call the VPAgent via a SUBSTEP() function to retrieve specific values. A Python reasoning engine then executes the code, ensuring that final calculations are numerically accurate and free from the "hallucinations" common in pure language models.
Experimental Results
The performance of VProChart was evaluated across three major CQA benchmarks: ChartQA (human-curated and machine-generated), PlotQA (large-scale complex reasoning), and DVQA (synthetic chart understanding).
VProChart established new state-of-the-art performance levels:
- ChartQA-Human: Achieved 57.4% accuracy, a significant 7.3% improvement over the previous best model.
- PlotQA: Reached 94.7% on the D1 subset and 83.5% on D2, outperforming competitors like MatCha and Pix2Struct.
- Ablation Studies: The inclusion of programmatic reasoning boosted performance on ChartQA-Human by 9.3%, confirming that code-based decomposition is essential for complex questions.
Discussion and Conclusion
VProChart demonstrates that explicitly modeling human-like visual rules—such as proximity and intersection—is more efficient than brute-force learning from massive datasets. By adopting a "Program-of-Thought" approach, the model separates logical reasoning from visual perception, allowing the LLM to handle logic while the VPAgent handles perception.
While the framework is highly effective, it remains dependent on the LLM's ability to generate valid code and requires more computational resources than single end-to-end models. Future work will focus on extending these perception capabilities to more complex schematic diagrams beyond standard bar and line charts.
Summary updated