Abstract
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in natural image understanding but continue to struggle with "text-rich" visual systems such as complex charts. We introduce ChartM³, a multi-stage, code-driven pipeline designed to bridge the gap between simple chart benchmarks and the sophisticated visual reasoning required for real-world business analytics. By generating synthetic data through executable Python code and Chain-of-Thought (CoT) reasoning, we produced a massive dataset of 142K Q&A pairs covering 62 chart types. Our results demonstrate that ChartM³ significantly boosts model performance in complex chart comprehension and cross-domain generalization, while also serving as a rigorous benchmark for state-of-the-art MLLMs.
Introduction
While models like GPT-4o excel at general visual tasks, they often fail when faced with professional-grade charts. Current benchmarks like ChartQA or PlotQA are limited by their focus on simple bar or line graphs and basic data retrieval. Real-world analytics, however, demands the comprehension of complex formats (e.g., heatmaps, box plots), multi-step reasoning (e.g., trend analysis, cross-data calculation), and interpretability. Existing datasets are often insufficient in scale or lack the detailed reasoning paths necessary for training sophisticated models.
The ChartM³ Pipeline
To address these gaps, we developed an automated, four-stage Multi-Stage Code-Driven Generation pipeline. This workflow ensures data accuracy and logical consistency by using code as the "source of truth."
- Template Database Construction: We curated 62 scientifically rigorous chart types across 60 diverse domains, including finance and biology.
- Chart Image Generation: Utilizing Retrieval-Augmented Generation (RAG), LLMs select professional templates and write Python code to simulate realistic data distributions and render high-quality charts.
- Instruction Q&A Generation: MLLMs formulate questions and synthesize analytical code. Executing this code produces 100% accurate ground-truth answers and detailed CoT reasoning paths.
- Data Evaluation: A multi-model verification process filters out low-quality outputs, ensuring visual clarity and logical integrity.
Dataset and Benchmark Characteristics
The resulting ChartM³ dataset is characterized by its unprecedented diversity and complexity. The training set comprises 38.4K charts and 142K high-quality Q&A pairs, while the benchmark includes 2,871 rigorously verified samples.
The dataset covers 9 major categories and 62 subcategories, ranging from standard bar charts to intricate "Combination" and "Progress" charts. This multi-dimensional approach ensures that models are exposed to a wide variety of visual structures and data representations.
Key Insights and Results
Our experiments yielded several critical insights into the state of chart comprehension in MLLMs:
- Code-Driven Verifiability: Using executable Python code eliminates the "hallucination" problem. If the code produces the answer, the data point is verifiable.
- The Impact of CoT: Training with Chain-of-Thought reasoning led to an 8% performance improvement on computation-heavy tasks compared to standard Q&A pairs.
- The Complexity Gap: Even top-tier models like Claude 3.5 Sonnet (66.18%) and GPT-4o (58.30%) struggle with the high-dimensional reasoning required by ChartM³, scoring significantly lower than on simpler benchmarks.
- Fine-Tuning and Generalization: A smaller model, Qwen2.5-VL-3B, when fine-tuned on ChartM³ using CoT-SFT, achieved a score of 62.88%, outperforming the base GPT-4o. Furthermore, models trained on ChartM³ showed significant improvements on other benchmarks like ReachQA (+7.6%) and CharXiv (+15%).
Conclusion
ChartM³ represents a significant step forward in constructing high-quality data for complex visual reasoning. By moving beyond simple retrieval to multi-step, code-verified analysis, we provide both a challenging benchmark and a powerful training resource. Future work will focus on expanding the pipeline to include other visualization languages and non-statistical diagrams like flowcharts.
Summary updated