ChartM3: A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehension

Duo Xu, Hao Cheng, Xin Lin, Zhen Xie, Hao Wang2025 · DOI

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in natural image understanding but continue to struggle with "text-rich" visual systems such as complex charts. We introduce ChartM³, a multi-stage, code-driven pipeline designed to bridge the gap between simple chart benchmarks and the sophisticated visual reasoning required for real-world business analytics. By generating synthetic data through executable Python code and Chain-of-Thought (CoT) reasoning, we produced a massive dataset of 142K Q&A pairs covering 62 chart types. Our results demonstrate that ChartM³ significantly boosts model performance in complex chart comprehension and cross-domain generalization, while also serving as a rigorous benchmark for state-of-the-art MLLMs.

Introduction

While models like GPT-4o excel at general visual tasks, they often fail when faced with professional-grade charts. Current benchmarks like ChartQA or PlotQA are limited by their focus on simple bar or line graphs and basic data retrieval. Real-world analytics, however, demands the comprehension of complex formats (e.g., heatmaps, box plots), multi-step reasoning (e.g., trend analysis, cross-data calculation), and interpretability. Existing datasets are often insufficient in scale or lack the detailed reasoning paths necessary for training sophisticated models.

The ChartM³ Pipeline

To address these gaps, we developed an automated, four-stage Multi-Stage Code-Driven Generation pipeline. This workflow ensures data accuracy and logical consistency by using code as the "source of truth."

  1. Template Database Construction: We curated 62 scientifically rigorous chart types across 60 diverse domains, including finance and biology.
  2. Chart Image Generation: Utilizing Retrieval-Augmented Generation (RAG), LLMs select professional templates and write Python code to simulate realistic data distributions and render high-quality charts.
  3. Instruction Q&A Generation: MLLMs formulate questions and synthesize analytical code. Executing this code produces 100% accurate ground-truth answers and detailed CoT reasoning paths.
  4. Data Evaluation: A multi-model verification process filters out low-quality outputs, ensuring visual clarity and logical integrity.
The ChartM³ pipeline automates the transition from template selection to code-driven data generation and quality verification.

Dataset and Benchmark Characteristics

The resulting ChartM³ dataset is characterized by its unprecedented diversity and complexity. The training set comprises 38.4K charts and 142K high-quality Q&A pairs, while the benchmark includes 2,871 rigorously verified samples.

The dataset covers 9 major categories and 62 subcategories, ranging from standard bar charts to intricate "Combination" and "Progress" charts. This multi-dimensional approach ensures that models are exposed to a wide variety of visual structures and data representations.

ChartM³ features 62 distinct chart types and evaluates models across multiple reasoning dimensions, highlighting current performance gaps.

Key Insights and Results

Our experiments yielded several critical insights into the state of chart comprehension in MLLMs:

  • Code-Driven Verifiability: Using executable Python code eliminates the "hallucination" problem. If the code produces the answer, the data point is verifiable.
  • The Impact of CoT: Training with Chain-of-Thought reasoning led to an 8% performance improvement on computation-heavy tasks compared to standard Q&A pairs.
  • The Complexity Gap: Even top-tier models like Claude 3.5 Sonnet (66.18%) and GPT-4o (58.30%) struggle with the high-dimensional reasoning required by ChartM³, scoring significantly lower than on simpler benchmarks.
  • Fine-Tuning and Generalization: A smaller model, Qwen2.5-VL-3B, when fine-tuned on ChartM³ using CoT-SFT, achieved a score of 62.88%, outperforming the base GPT-4o. Furthermore, models trained on ChartM³ showed significant improvements on other benchmarks like ReachQA (+7.6%) and CharXiv (+15%).

Conclusion

ChartM³ represents a significant step forward in constructing high-quality data for complex visual reasoning. By moving beyond simple retrieval to multi-step, code-verified analysis, we provide both a challenging benchmark and a powerful training resource. Future work will focus on expanding the pipeline to include other visualization languages and non-statistical diagrams like flowcharts.

Summary updated

BibTeX
@misc{xu2025chartm,
  author = {Duo Xu and Hao Cheng and Xin Lin and Zhen Xie and Hao Wang},
  title = {{ChartM3: A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehension}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.701},
}
Figure