Introduction
Chart Question Answering (CQA) represents a unique challenge at the intersection of visual perception and logical reasoning. While Multimodal Large Language Models (MLLMs) like GPT-4V have improved in "seeing" charts, they often struggle with complex, multi-step reasoning. Conversely, text-only Large Language Models (LLMs) possess superior reasoning capabilities but are "blind" to visual data, relying on lossy text-to-table conversions.
Researchers from the Chinese Academy of Sciences have introduced SYNERGY, a novel framework that treats LLMs as the "brain" and MLLMs as the "eyes" to address these bottlenecks. By decomposing complex queries into manageable sub-tasks and augmenting visual inputs with structured data, SYNERGY achieves state-of-the-art performance without additional model training.
The SYNERGY Framework
SYNERGY operates through a modular four-stage pipeline designed to maximize the strengths of both LLMs and MLLMs.
- Perceptual Signal Augmentation: To provide both intuitive context and precise data, SYNERGY uses a
chart2tablemodel (e.g., DEPLOT) to generate a JSON-formatted table. This "augmented perceptual signal" (au-signal) combines the original chart image with structured numerical data. - Question Decomposition: A "Decomposer" LLM breaks a complex query into atomic sub-questions. By referencing table headers, the LLM ensures these sub-questions remain grounded in the chart's content.
- Perception Enhancement: An MLLM (the "Perceptual Reasoner") answers these simpler sub-questions sequentially. Reducing task complexity significantly lowers the likelihood of reasoning errors or hallucinations.
- Summary Verification: A "Verifier" LLM synthesizes the final answer by reviewing the sub-question/answer pairs and the original reasoning path, correcting any inconsistencies.
Experimental Evaluation
The framework was evaluated on two primary benchmarks: ChartQA, which includes complex human-written questions, and PlotQA-sub, designed to test both content perception and numerical reasoning.
Key Results
- Performance Gains: SYNERGY provided a massive boost to open-source models. For instance, LLaVA-V1.5 (13B) saw a +47.46% increase in average accuracy on ChartQA.
- State-of-the-Art: Using GPT-4 as the "brain," SYNERGY achieved a superior accuracy of 67.20% on the ChartQA human-written set, outperforming specialized models like UniChart.
- Model Agnostic: The framework demonstrated high compatibility across various LLMs (LLaMA-3, Gemini-Pro, GPT-4) and MLLMs (LLaVA, Qwen-VL). Even smaller 8B parameter models like LLaMA-3 proved effective as decomposers and verifiers.
Discussion and Conclusion
The success of SYNERGY highlights that tables and charts are complementary: tables provide precise numerical data, while charts offer intuitive visual cues. By offloading complex logic to LLMs and focusing MLLMs on direct perception, the framework overcomes the "fragile reasoning" bottleneck of current multimodal systems.
While SYNERGY introduces higher inference costs due to its multi-stage nature, it sets a new standard for training-free CQA. Future work will focus on improving inference efficiency and exploring end-to-end synergistic training methods.
Summary updated