TL;DR
M³OPD-Reasoner introduces a robust, three-stage "recipe" for post-training vision-language models (VLMs) to excel at complex reasoning across image STEM, perception, and video domains. By combining specialized domain teachers through Multi-Teacher On-Policy Distillation (MOPD) and then refining the result with a final round of Reinforcement Learning (RL), the authors create a 9B-parameter model that matches or exceeds specialized experts while remaining a versatile generalist.
What Problem Does This Paper Address?
Training a single VLM to be a "master of all trades" in reasoning is notoriously difficult. When you try to train one model on a mix of diverse data—like calculus problems, chart reading, and video sequence analysis—you often run into gradient conflict. Improving the model's performance in one domain (e.g., math) might inadvertently degrade its performance in another (e.g., video understanding).
Existing "recipes" usually rely on Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) on mixed data. However, this often fails to fully capture the specialized reasoning capabilities that individual domain-specific models can achieve. The authors sought a way to unify these specialized "expert" skills into a single student model without the typical performance trade-offs.
Methods
The authors propose a three-stage post-training pipeline, starting from a Qwen3.5-9B base:
- Domain Teachers: They first train three separate "master" models, one for each domain (Image STEM, Image Perception, and Video). These teachers are trained using Group Sequence Policy Optimization (GSPO), a form of RL that uses rule-based verifiable rewards (e.g., checking if a math answer is correct).
- Multi-Teacher On-Policy Distillation (MOPD): Instead of just mixing data, they use the three masters to teach a single student. The student generates its own responses (on-policy), and the corresponding domain teacher "corrects" it by providing token-level log-probabilities. This allows the student to absorb the specialized reasoning styles of all three teachers.
- MixRL on top of MOPD: In a novel final step, they take the distilled student and perform another round of RL on the combined dataset. Because MOPD has already successfully merged the domain capabilities, this final RL stage starts from a much stronger baseline and can push the model's reasoning even further.
Datasets
The authors curated and released two massive, decontaminated datasets:
- M³OPD-SFT-1.35M: A collection of 1.35 million samples for supervised fine-tuning, using "self-traces" (reasoning chains generated by the model itself) which were found to be more effective for distillation than traces from larger, external models.
- M³OPD-RL-193K: 193,000 prompts with rule-verifiable answers (like multiple-choice or exact numeric values) used for the RL stages.
Key Insights and Ideas
- MOPD as a "Stronger Starting Point": The most significant finding is that distillation is a better way to merge domain skills than joint RL training. By merging skills first via MOPD, the subsequent MixRL stage doesn't have to "fight" to balance domains; it can focus purely on sharpening reasoning.
- The "Self-Trace" Advantage: Surprisingly, training a student on its own correct reasoning chains (self-traces) often yields better results than training on traces from a much more powerful model. This is likely because the self-generated traces are more compatible with the student's internal distribution.
- No Collapse in Multi-Stage RL: Unlike some previous findings where multiple RL stages lead to model "collapse" or over-optimization, the authors show that MixRL on top of a distilled student consistently improves performance across almost all benchmarks.
- Verifiable Rewards are Key: By sticking to rule-based rewards (accuracy + format), the researchers avoid the "reward hacking" often seen with neural-network-based reward models in reasoning tasks.
Key Results
The final model, M³OPD-Reasoner-9B, demonstrates substantial improvements over the base model and standard mixed-training baselines:
- Overall Gains: It achieved an average accuracy of 75.9% across 21 benchmarks, compared to 69.4% for the base Qwen3.5-9B.
- Domain Excellence: It saw massive jumps in specific areas: +7.1% in Image STEM, +3.0% in Image Perception, and a whopping +10.7% in Video reasoning.
- Competitive Performance: On many benchmarks, this 9B model outperforms much larger models or those specifically built for single domains (like Video-MOPD).
Limitations and Future Work
- Verifiable Reward Constraint: The current recipe relies on tasks with verifiable answers (math, MCQ, etc.). Extending this to open-ended reasoning where "correctness" is harder to define remains a challenge.
- Model Scale: The study focused on the 9B parameter scale. While highly effective, exploring how this recipe scales to 70B+ parameter models is a natural next step.
- Reasoning Quality: While the model gets the right answers, the "thinking" traces themselves aren't directly rewarded for elegance or conciseness—only for leading to the correct final answer. Future work could involve more granular rewards for the reasoning process itself.
Summary updated