MindGYM: What Matters in Question Synthesis for Thinking-Centric Fine-Tuning?

Zhe Xu, Daoyuan Chen, Zhenqing Ling, Yaliang Li, Ying Shen

TL;DR

MINDGYM is a novel data synthesis framework that enables large vision-language models (VLMs) to self-generate high-quality, cognitively diverse multi-hop reasoning data. By fine-tuning models on just 400 of these structured, self-generated samples, MINDGYM achieves significant performance boosts across multiple text and multimodal reasoning benchmarks, outperforming baselines trained on up to ten times more data.

What Problem Does This Paper Address?

While large foundation models excel at many tasks, training them to possess robust, transferable reasoning abilities remains a major challenge. Current instruction-tuning approaches typically rely on two paradigms, both of which have bottlenecks:

  1. Manual curation: Datasets like OK-VQA or ScienceQA are high-quality but labor-intensive and expensive to scale.
  2. Automated synthesis: Methods like MMEvol or MMInstruct scale easily but often produce shallow, repetitive, or logically inconsistent data.

Furthermore, alternative methods like reinforcement learning (RL) or iterative prompting incur prohibitive computational costs. MINDGYM addresses this gap by introducing a data-centric, self-challenging paradigm: enabling models to systematically target their own cognitive bottlenecks by generating structured, multi-hop questions.

Methods

MINDGYM breaks down question synthesis into a structured four-stage pipeline that injects human-like cognitive traits (breadth, depth, and difficulty progression) directly into the prompt design.

Figure 1: The proposed MINDGYM framework incorporates a cognitively guided data synthesis pipeline with four stages: ①Context Generation, ②Single-Hop Question Synthesis, ③Multi-Hop Composition with Thinking Trace, and ④Structured Extraction. Starting from a meta-topic and guided by a shared cognitive objective, the model iteratively builds background context, atomic reasoning steps, composite questions with interpretable thinking traces, and final structured QA samples for downstream use. Color-coded regions and arrows illustrate the hierarchical progression from simple reasoning to advanced problem-solving and emphasizing conceptual depth.

The framework consists of three core components:

  • Cognitive Thinking Process Injection: Adapting protocols like Thinking-Claude, the framework prompts the model to externalize its internal stream of consciousness (e.g., hypothesis formulation, counterfactual reasoning, self-verification) before generating data.
  • Seed Single-Hop Question Synthesis: The model generates atomic, single-hop question-answer primitives grounded in a self-generated background context across eight meta-topics (e.g., causal inference, economic principles). A reject sampling mechanism filters out semantically redundant questions using cosine similarity.
  • Challenging Multi-Hop QA Synthesis: The model uses thinking-centric operators—such as Bridging, Comparison, and Temporal reasoning—to compose the atomic seeds into complex, multi-hop questions with explicit reasoning trajectories.

Finally, instead of uniform training, MINDGYM employs a Structured Progression Training pathway. The model transitions from guided answering (receiving the question and reasoning trace) to reason reconstruction, paired reasoning, and ultimately autonomous solving, mirroring human cognitive development.

Datasets

The authors evaluated MINDGYM across six prominent reasoning benchmarks spanning two modalities:

  • Text-Based Benchmarks: GSM8K (grade-school math), MATH (competition-level math), and GPQA (graduate-level Google-proof Q&A).
  • Multimodal Benchmarks: MMStar, MathVista-Mini, and MathVision-Mini.

MINDGYM's 400-sample synthetic datasets were compared against established SOTA text and multimodal datasets, including LIMO (817 curated logical chains), Open-O1 (subsampled to 400 and 4,000 samples), and core subsets of MMEvol (ScienceQA and DvQA).

Key Insights and Ideas

  • Quality Variance is a Silent Killer: Using the Data-Juicer toolkit, the authors discovered that high average data quality is only half the battle. Reducing quality variance (standard deviation) is absolutely critical for stable fine-tuning. Larger models naturally generate synthetic data with significantly lower variance, making them excellent data synthesizers.
  • Structure Trumps Factual Perfection: In an ablation study, removing data flagged as "logically flawed" or "hallucinatory" by GPT-4o actually degraded downstream model performance. This suggests that for thinking-centric tuning, the diversity and structure of the reasoning trajectories matter more than strict factual precision.
  • Cross-Modal Transferability: Even when MINDGYM synthesized data in a pure-text format, it significantly boosted performance on vision-intensive benchmarks like MathVision. Strengthening the model's intrinsic, abstract thinking capabilities naturally propagates improvements across modalities without explicit visual supervision.

Key Results

Despite using a tiny fraction of the data volume used by traditional pipelines, MINDGYM achieved remarkable results across different model architectures (Qwen and InternVL series) and scales (7B to 38B parameters):

Table 1: Performance of different models on various evaluation benchmarks. The number following the dataset name indicates its size in terms of the number of samples used for training.
  • Massive Efficiency Gains: With only 400 synthetic samples, MINDGYM boosted the performance of Qwen2.5-VL-7B on MathVision-Mini by over 16%, outperforming Open-O1 baselines trained on 10× more data.
  • Superior Data Properties: Data-Juicer analysis showed that MINDGYM's synthetic data achieved a 16.7% higher average quality and a 67.91% reduction in quality variance compared to baseline sources.
  • Win-Rate Dominance: Evaluated by GPT-4 as an expert scorer, MINDGYM achieved an overall win-rate improvement of 15.8% in thinking depth and breadth over competitive baselines.

Limitations and Future Work

The authors note that extending MINDGYM to fully multimodal synthesis remains an open challenge. Currently, generating text-image QA pairs relies on static anchor image datasets (like ScienceQA) to prevent the model from hallucinating visual elements.

To achieve true self-evolution, future work will focus on:

  1. Incorporating richer, more diverse image datasets.
  2. Developing fully generative pipelines where the VLM synthesizes both the image and the associated reasoning task simultaneously.
  3. Expanding cognitive relationships into dynamic visual scenarios (video) and specialized domains like medicine and finance.

Summary updated

Abstract

Large foundation models face challenges in acquiring transferable, structured thinking abilities, especially when supervised with rigid templates or crowd-annotated instruction datasets. Unlike prior approaches, we focus on a thinking-centric data synthesis paradigm that enables models to evolve through self-generated, cognitively guided data. We propose MindGYM, a structured and scalable framework for question synthesis, composed of: (1) Cognitive Thinking Process Injection, which infuses high-level reasoning objectives to shape the model's synthesis behavior; (2) Seed Single-Hop Question Synthesis, generating atomic questions from diverse semantic types to encourage broader thinking; and (3) Challenging Multi-Hop QA Synthesis, composing more complex multi-hop questions based on QA seeds for deeper reasoning. Detailed analysis shows that synthetic data generated by our method achieves 16.7% higher average quality and 67.91% lower quality variance compared to baseline sources, highlighting that both high-quality and self-contained data are essential for effective, thinking-oriented fine-tuning. MindGYM improves performance on six reasoning benchmarks, achieving gains of up to 16% on MathVision using only 400 data samples, and generalizable improvements across different model sizes and architectures. MindGYM underscores the viability of self-challenging mechanisms in refining large model capabilities while minimizing human intervention and resource demands. Code and data are released to promote data-centric research into self-evolving foundation models driven by their internal reasoning capabilities.
BibTeX
@inproceedings{xu2025mindgym,
  author = {Zhe Xu and Daoyuan Chen and Zhenqing Ling and Yaliang Li and Ying Shen},
  title = {{MindGYM: What Matters in Question Synthesis for Thinking-Centric Fine-Tuning?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=J6Is1CdvB8},
}
Figure