Abstract
Reinforcement learning (RL) with verifiable rewards has recently catalyzed the development of "MLLM-r1" approaches, bringing advanced reasoning capabilities to vision language models. Most paradigms begin with a "cold start" phase—typically supervised fine-tuning (SFT)—to initialize the policy before RL. However, SFT-based cold starts often lead to instruction-style overfitting, where the model intertwines reasoning with specific output formats, weakening out-of-distribution generalization. We introduce SPECS (Self-distilled, Preference-based Cold Start), a framework that decouples multimodal learning by focusing on surface-form criteria through preference-based training before handing off to RL for deep reasoning. Our approach improves performance on MathVista by 12.2% and MEGA-Bench by 4.1%, while enhancing training stability and exploration.
Introduction
The emergence of models like DeepSeek-R1 has demonstrated that Reinforcement Learning (RL) can significantly enhance the reasoning capabilities of large models. In the multimodal domain, this usually involves a cold start phase where the model is fine-tuned on high-quality Chain-of-Thought (CoT) data.
Despite its popularity, the authors identify a critical flaw in SFT-based cold starts: instruction-style overfitting. When a model learns reasoning logic and output formatting (e.g., <think> tags) simultaneously via SFT, it tends to memorize specific task-solution pairs. This "stuckness" limits the model's ability to explore diverse reasoning paths during the subsequent RL phase, ultimately capping its performance ceiling.
The SPECS Framework
To address these limitations, we propose the SPECS framework. SPECS decouples the learning process into three distinct stages:
1. Self-Distillation for Data Generation
Rather than relying on expensive human annotations or external teacher models (like GPT-4o), SPECS uses a brief, initial RL phase (GRPO) to create a "seed model." This model generates responses to training prompts. To create preference pairs, the authors "pollute" the format of correct responses—for example, by removing required tags or altering the structure—while keeping the factual content intact. This results in a dataset where both "chosen" and "rejected" answers are factually correct, but only the "chosen" one adheres to the desired format.
2. Preference-based Pre-Alignment (DPO)
The model undergoes Direct Preference Optimization (DPO) on these self-distilled pairs. Because both options in a pair are correct, the model cannot rely on factual correctness to minimize loss. Instead, it is forced to focus on "shallow," transferable surface-form features such as structure, style, and formatting. This prevents the model from over-optimizing on specific reasoning content early on.
3. Final Reinforcement Learning
With the formatting mastered, the model enters the full RL phase using Group Relative Policy Optimization (GRPO) with verifiable rewards. Because the "how to speak" (format) is already settled, the RL phase can focus entirely on "what to think" (deep reasoning).
Quantifying Generalization
To evaluate the effectiveness of different cold start methods, the researchers introduced the Generalization Factor (GF). Based on the $F_\beta$-score, the GF quantifies a model's ability to improve performance across both in-distribution and out-of-distribution tasks. Empirical studies using this metric revealed that preference-based methods (DPO) consistently generalize better than likelihood-based methods (SFT) during the cold start phase.
Experimental Results
SPECS was evaluated against strong baselines, including Qwen2.5-VL-7B, across several rigorous multimodal benchmarks:
- MathVista: Achieved a 12.2% improvement over the baseline.
- MEGA-Bench: Observed a 4.1% increase in core scores.
- Training Stability: Qualitative analysis showed that SPECS leads to smoother RL training curves. SFT-based starts often exhibit high volatility and "stuckness," whereas SPECS maintains a higher Rollout Branching Factor (RBF), allowing the model to remain creative and explore more diverse reasoning paths.
Conclusion
Metis-SPECS demonstrates that the method of initialization is just as critical as the RL phase itself. By decoupling format learning from reasoning through self-distilled preference optimization, SPECS provides a more robust foundation for VLMs. This approach not only improves benchmark performance but also ensures that models remain flexible and capable of complex exploration in unseen domains. Future work will focus on extending this framework to text-only reasoning and larger-scale parameter models.
Summary updated