Introduction
While Large Multimodal Models (LMMs) have advanced general image understanding, they frequently falter when interpreting charts. This difficulty stems from two primary challenges: perceptual complexity and reasoning complexity. Charts lack universal standards, requiring models to implicitly structure spatial relationships and semantic cues. Furthermore, multi-step logical or numerical operations often lead to "hallucinations" in purely neural architectures. VProChart addresses these hurdles by integrating a lightweight visual perception agent with a programmatic reasoning engine.
Methodology
The VProChart framework splits the Chart Question Answering (CQA) task into two specialized modules designed to handle perception and logic independently.
Visual Perception Alignment Agent (VPAgent)
The VPAgent treats charts not as flat pixel grids but as structured collections of elements. It employs three "Human Visual Perception" principles—Crosshair Intersecting, Spatial Proximity, and Color/Semantic Consistency—to align data points with their corresponding axes and labels. By using a Swin-Encoder for visual features and a Q-driven reasoning module, the agent focuses specifically on clusters relevant to the user's question.
Programmatic Solution Reasoning
To ensure mathematical and logical precision, VProChart moves away from "black-box" neural guessing. It utilizes a Large Language Model (LLM), such as Qwen2-7B-Instruct, to act as a programmer. The LLM parses natural language questions into Python-style programs that use a SUBSTEP() function to query the VPAgent for specific data points. A Python engine then executes these programs, ensuring that if the data extraction is accurate, the final calculation is perfect.
Evaluation and Key Results
VProChart was evaluated on three major benchmarks: ChartQA, PlotQA, and DVQA. The framework achieved state-of-the-art (SOTA) performance across these datasets. Notably, it reached 57.4% accuracy on the ChartQA-Human subset, nearly doubling the performance of earlier large-scale models like PaLI. These results suggest that explicit modeling of human perceptual biases is more effective for real-world charts than relying solely on synthetic pre-training.
Conclusion and Future Work
By separating perception from computation, VProChart avoids the "fuzzy math" problem common in standard LMMs. While the system currently depends on the LLM's ability to generate correct code and introduces some latency due to its two-stage process, it represents a significant step toward human-like chart understanding. Future research will explore expanding the framework to more complex schematic diagrams beyond standard bar and line charts.
Summary updated