Training Multimodal Large Reasoning Models Needs Better Thoughts: A Three-Stage Framework for Long Chain-of-Thought Synthesis and Selection

Yizhi Wang, Linan Yue, Min-Ling ZhangarXiv 2025 · DOI · arXiv

Introduction

While text-only models like DeepSeek-R1 have achieved remarkable success in "slow thinking" through long Chain-of-Thought (CoT) reasoning, Multimodal Large Reasoning Models (MLRMs) have struggled to keep pace. The primary obstacle is a severe "data drought": high-quality thinking traces for visual problems are rare. Existing datasets often feature reasoning steps that are too brief, suffer from modality errors where visual nuances are lost in text conversion, or rely on rigid, robotic synthetic prompts.

To address these challenges, we introduce SynSelect, a three-stage framework designed to synthesize and select high-quality, flexible, and instructive multimodal reasoning data.

The SynSelect Framework

SynSelect operates through a pipeline that prioritizes architectural diversity and rigorous validation to produce superior training signals.

I. Reasoning Trajectory Synthesis

Rather than relying on a single model, SynSelect utilizes a "committee" of heterogeneous MLRMs, termed SynAgents. These agents generate multiple diverse reasoning paths for each question. This architectural diversity ensures a broad range of reasoning styles and "aha moments," avoiding the repetitive nature of single-model synthesis.

II. Instance-Level Selection

To identify the most effective reasoning paths, SynSelect applies a hierarchical filtering process:

  1. Answer Correctness: Only trajectories leading to the correct final answer are retained.
  2. The "LLMPlayer" Strategy: This novel "weak-to-strong" validation uses a small, "weak" model (e.g., Qwen2.5-VL-3B) as a litmus test. If the weak model can only solve a difficult problem when provided with a specific CoT, that CoT is deemed highly instructive.
  3. Rationale Ratio: An LLM-based extractor calculates the ratio of meaningful rationale to total length, ensuring the "thought" is dense with information rather than filled with meaningless repetitions.

III. Batch Selection

Recognizing that not all questions are equally valuable for training, SynSelect prioritizes queries where the synthesized CoT provides the greatest "Accuracy Gain" and "Confidence Gain." By focusing on non-trivial problems, the framework can achieve better results using only a refined subset (e.g., 20%) of the total data.

Key Results and Insights

SynSelect demonstrates that quality and instructiveness are more important than raw data volume. Models fine-tuned on SynSelect data consistently outperformed state-of-the-art baselines. On the MathVista benchmark, SynSelect-SFT achieved 64.3% accuracy, a significant jump from the R1-Onevision baseline of 59.9%. When combined with Reinforcement Learning (GRPO), performance further climbed to 66.0%.

Beyond accuracy, SynSelect produces more efficient reasoning. As shown in the comparison between synthesized and original data, SynSelect trajectories are often significantly shorter (e.g., 217 tokens vs. 494 tokens) while remaining more accurate and less redundant. Furthermore, because the framework uses native MLRMs for synthesis, it preserves spatial and visual nuances—such as geometric notches—that are typically lost in traditional image-to-text pipelines.

Conclusion and Future Work

SynSelect provides a robust path forward for training the next generation of multimodal reasoners. By leveraging weak-to-strong validation and iterative bootstrapping, it overcomes the limitations of existing synthetic datasets. Future work will explore extending these selection metrics to include "modality alignment" scores, ensuring models are truly grounding their reasoning in specific visual elements during every step of the thought process.

Summary updated

Abstract

Large Reasoning Models (LRMs) have demonstrated remarkable performance on complex reasoning tasks through long Chain-of-Thought (CoT) reasoning. Extending these successes to multimodal reasoning remains challenging due to the increased complexity of integrating diverse input modalities and the scarcity of high-quality long CoT training data. Existing multimodal datasets and CoT synthesis methods still suffer from limited reasoning depth, modality conversion errors, and rigid generation pipelines, hindering model performance and stability. To this end, in this paper, we propose SynSelect, a novel three-stage Synthesis-Selection framework for generating high-quality long CoT data tailored to multimodal reasoning tasks. Specifically, SynSelect first leverages multiple heterogeneous multimodal LRMs to produce diverse candidate CoTs, and then applies both instance and batch level selection to filter high-quality CoTs that can effectively enhance the model's reasoning capabilities. Extensive experiments on multiple multimodal benchmarks demonstrate that models supervised fine-tuned on SynSelect-generated data significantly outperform baselines and achieve further improvements after reinforcement learning post-training. Our results validate SynSelect as an effective approach for advancing multimodal LRMs reasoning capabilities.
BibTeX
@misc{wang2025training,
  author = {Yizhi Wang and Linan Yue and Min-Ling Zhang},
  title = {{Training Multimodal Large Reasoning Models Needs Better Thoughts: A Three-Stage Framework for Long Chain-of-Thought Synthesis and Selection}},
  year = {2025},
  eprint = {2512.18956},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.18956},
}
Figure