TL;DR
This paper introduces "Chain of Functions" (CoF), a novel programmatic pipeline for generating high-quality, diverse, and explainable reasoning data for multimodal large language models (MLLMs) tackling complex chart queries. By leveraging function chains as programmatic supervision, CoF overcomes the limitations of direct prompting, leading to more precise data and enabling fine-grained evaluation of MLLMs.
What Problem Does This Paper Address?
Multimodal Large Language Models (MLLMs) are increasingly capable of understanding visual information, but they often struggle with complex chart reasoning tasks. A key bottleneck is the scarcity of high-quality "chain-of-thought" (CoT) data, which includes detailed rationales for how an answer is derived from a chart. Existing methods that use LLMs to generate this data often suffer from limited precision and diversity due to the inherent nature of freeform prompting. Furthermore, current evaluation metrics for MLLMs on chart reasoning are often coarse, lacking the granularity to pinpoint specific weaknesses. This paper aims to address this gap by proposing a more robust and scalable method for generating CoT data that facilitates both improved MLLM performance and detailed evaluation.
Methods
The core innovation is the "Chain of Functions" (CoF) pipeline, which consists of two main stages:
- Program-based Functional Discovery: Instead of direct prompting, CoF first explores a set of predefined "atomic functions" (e.g., selecting data points, performing arithmetic operations, finding maximum values) in a human-free manner. These functions are combined to form diverse "function chains" that represent programmatic reasoning paths for a given chart. This programmatic approach ensures precision and diversity by systematically enumerating possible reasoning steps.
- Reverse Linguistic CoT Data Synthesis: The generated function chains are then translated into natural language CoT data (rationales and questions) using a moderate-sized, open-sourced LLM. This "reverse" approach, where the program dictates the linguistic output, significantly reduces hallucinations and ensures the generated rationales are grounded in the programmatic logic. This also makes the process less reliant on extremely large, proprietary LLMs.
Datasets
The authors introduce ChartCoF, a new dataset designed for fine-grained chart reasoning evaluation and enhancement.
- ChartCoF: This dataset comprises 19 chart types, including common ones (bar, line, pie) and more complex ones (radar, candlestick, treemap, etc.). It contains 18,349 charts with 50,329 Q&A pairs for training and 648 charts with 1,451 Q&A pairs for testing.
- Existing Datasets: The paper also evaluates MLLMs on existing benchmarks like ChartQA, ChartBench, and ChartX to demonstrate the effectiveness of their proposed method.
Key Insights and Ideas
- Programmatic Supervision for Precision: The key insight is that using programmatic function chains as supervision, rather than freeform text prompts, significantly improves the precision of generated reasoning data. This is because the underlying logic is explicitly defined and executed.
- Diversity through Exploration: By systematically exploring combinations of atomic functions, CoF generates a wide variety of reasoning paths, leading to more diverse questions and rationales than template-based methods.
- Explainability as a Feature: The function chains themselves serve as built-in rationales, offering a transparent view of the model's reasoning process. This enables fine-grained evaluation beyond simple accuracy metrics.
- Democratizing Data Generation: CoF's reliance on moderate LLMs for linguistic transfer makes high-quality data generation more accessible, reducing the dependency on massive, proprietary models.
- Fine-grained Evaluation is Crucial: The paper highlights that existing MLLMs exhibit significant performance disparities across different question taxonomies and reasoning complexities, underscoring the need for fine-grained evaluation.
Key Results
- ChartCoF Dataset: The proposed CoF pipeline successfully generated the ChartCoF dataset, which is shown to be diverse and accurate.
- Performance Improvement: Finetuning MLLMs (e.g., InternVL-2.5-8B, Qwen2VL-7B) with ChartCoF significantly improved their accuracy on existing benchmarks (ChartBench, ChartQA, ChartX) and on ChartCoF itself. For instance, InternVL-2.5-8B saw an 8.56% improvement on ChartBench.
- Fine-grained Analysis: The fine-grained evaluation on ChartCoF revealed that MLLMs struggle with complex reasoning questions, with performance varying significantly based on function chain taxonomies and question lengths.
- Out-of-Distribution Generalization: Models trained with ChartCoF demonstrated improved generalization to unseen chart types and longer function chains, indicating robust reasoning capabilities.
- Rationale Refinement: An ablation study showed that refining generated rationales leads to more fluent and informative explanations, further enhancing MLLM performance.
Limitations and Future Work
- Chart Representation: The current approach uses JSON data to represent charts, which, despite efforts with code templates, may still differ from naturally occurring charts found online. Future work could explore more sophisticated methods for extracting information from web-based charts or improving chart rendering techniques.
- LLM Accuracy: While CoF reduces hallucinations, LLMs used for linguistic transfer might still produce occasional inaccuracies. Future research could focus on developing mechanisms to filter these out for even higher data quality.
- Broader Applications: The authors suggest that the programmatic function chain paradigm could be extended to other step-wise reasoning tasks beyond charts, such as mathematical Q&A or graphical user interface tasks.
Summary updated