M3OPD-Reasoner: An Open Recipe for Post-Training Multimodal Reasoning with Multi-Teacher On-Policy Distillation

Nguyen Quang Trung, Hoang Phan, Shuo Sun, Longyin Zhang, AiTi Aw

TL;DR

M³OPD-Reasoner introduces a robust, three-stage "recipe" for post-training vision-language models (VLMs) to excel at complex reasoning across image STEM, perception, and video domains. By combining specialized domain teachers through Multi-Teacher On-Policy Distillation (MOPD) and then refining the result with a final round of Reinforcement Learning (RL), the authors create a 9B-parameter model that matches or exceeds specialized experts while remaining a versatile generalist.

What Problem Does This Paper Address?

Training a single VLM to be a "master of all trades" in reasoning is notoriously difficult. When you try to train one model on a mix of diverse data—like calculus problems, chart reading, and video sequence analysis—you often run into gradient conflict. Improving the model's performance in one domain (e.g., math) might inadvertently degrade its performance in another (e.g., video understanding).

Existing "recipes" usually rely on Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) on mixed data. However, this often fails to fully capture the specialized reasoning capabilities that individual domain-specific models can achieve. The authors sought a way to unify these specialized "expert" skills into a single student model without the typical performance trade-offs.

Methods

The authors propose a three-stage post-training pipeline, starting from a Qwen3.5-9B base:

  1. Domain Teachers: They first train three separate "master" models, one for each domain (Image STEM, Image Perception, and Video). These teachers are trained using Group Sequence Policy Optimization (GSPO), a form of RL that uses rule-based verifiable rewards (e.g., checking if a math answer is correct).
  2. Multi-Teacher On-Policy Distillation (MOPD): Instead of just mixing data, they use the three masters to teach a single student. The student generates its own responses (on-policy), and the corresponding domain teacher "corrects" it by providing token-level log-probabilities. This allows the student to absorb the specialized reasoning styles of all three teachers.
  3. MixRL on top of MOPD: In a novel final step, they take the distilled student and perform another round of RL on the combined dataset. Because MOPD has already successfully merged the domain capabilities, this final RL stage starts from a much stronger baseline and can push the model's reasoning even further.
The M³OPD-Reasoner recipe as a four-part story: training masters, distillation, practice/check, and the final generalist model.
Detailed technical overview of the three-stage pipeline: (1) Domain-specific GSPO, (2) Multi-teacher distillation, and (3) Final MixRL refinement.

Datasets

The authors curated and released two massive, decontaminated datasets:

  • M³OPD-SFT-1.35M: A collection of 1.35 million samples for supervised fine-tuning, using "self-traces" (reasoning chains generated by the model itself) which were found to be more effective for distillation than traces from larger, external models.
  • M³OPD-RL-193K: 193,000 prompts with rule-verifiable answers (like multiple-choice or exact numeric values) used for the RL stages.
Distribution of data sources across the three reasoning domains for both SFT and RL stages.

Key Insights and Ideas

  • MOPD as a "Stronger Starting Point": The most significant finding is that distillation is a better way to merge domain skills than joint RL training. By merging skills first via MOPD, the subsequent MixRL stage doesn't have to "fight" to balance domains; it can focus purely on sharpening reasoning.
  • The "Self-Trace" Advantage: Surprisingly, training a student on its own correct reasoning chains (self-traces) often yields better results than training on traces from a much more powerful model. This is likely because the self-generated traces are more compatible with the student's internal distribution.
  • No Collapse in Multi-Stage RL: Unlike some previous findings where multiple RL stages lead to model "collapse" or over-optimization, the authors show that MixRL on top of a distilled student consistently improves performance across almost all benchmarks.
  • Verifiable Rewards are Key: By sticking to rule-based rewards (accuracy + format), the researchers avoid the "reward hacking" often seen with neural-network-based reward models in reasoning tasks.

Key Results

The final model, M³OPD-Reasoner-9B, demonstrates substantial improvements over the base model and standard mixed-training baselines:

  • Overall Gains: It achieved an average accuracy of 75.9% across 21 benchmarks, compared to 69.4% for the base Qwen3.5-9B.
  • Domain Excellence: It saw massive jumps in specific areas: +7.1% in Image STEM, +3.0% in Image Perception, and a whopping +10.7% in Video reasoning.
  • Competitive Performance: On many benchmarks, this 9B model outperforms much larger models or those specifically built for single domains (like Video-MOPD).
Performance gains of the MOPD and final Reasoner models over the base Qwen3.5-9B across various benchmarks.

Limitations and Future Work

  • Verifiable Reward Constraint: The current recipe relies on tasks with verifiable answers (math, MCQ, etc.). Extending this to open-ended reasoning where "correctness" is harder to define remains a challenge.
  • Model Scale: The study focused on the 9B parameter scale. While highly effective, exploring how this recipe scales to 70B+ parameter models is a natural next step.
  • Reasoning Quality: While the model gets the right answers, the "thinking" traces themselves aren't directly rewarded for elegance or conciseness—only for leading to the correct final answer. Future work could involve more granular rewards for the reasoning process itself.

Summary updated

Abstract

Multi-Teacher On-Policy Distillation (MOPD) enables a single student to absorb the skills of several domain teachers by learning from its own generated responses. We present Multimodal Model Multi-Teacher On-Policy Distillation Reasoner (M3OPD-Reasoner), a fully open recipe for post-training vision-language models (VLMs) over three reasoning domains: image STEM, image perception, and video. We curate two datasets, M3OPD-SFT-1.35M for supervised fine-tuning (SFT) and M3OPD-RL-193K for reinforcement learning (RL) with verifiable rewards. On these data, we compare post-training recipes that start from Qwen3.5-9B: training one model on the mixed data of all domains (MixSFT and MixRL), and training one teacher per domain with Group Sequence Policy Optimization (GSPO), directly or after an SFT warm-up, and distilling the teachers into one student with MOPD. We evaluate on 21 image STEM, image perception, and video reasoning benchmarks after we decontaminate the training data against them. The comparison establishes MOPD of teachers trained with GSPO directly from Qwen3.5-9B as the best recipe: the student closely matches each teacher on its own domain and is on average above the mixed-training baselines on the same prompts. Since MOPD already merges the domain capabilities that MixRL would otherwise have to learn, its student is a stronger starting point for RL, and we ask whether the same data can carry the model further from there: we add MixRL on top of the MOPD student, on the same prompts and with the same settings, which, to our knowledge, no prior VLM recipe has studied. The same prompts still improve the MOPD student, with no collapse, and give our main model, M3OPD-Reasoner-9B. MOPD and MixRL can therefore work as consecutive stages that get more out of the same data. We release all data, the training code, and the checkpoints of every stage.
BibTeX
@misc{trung2026m,
  author = {Nguyen Quang Trung and Hoang Phan and Shuo Sun and Longyin Zhang and AiTi Aw},
  title = {{M3OPD-Reasoner: An Open Recipe for Post-Training Multimodal Reasoning with Multi-Teacher On-Policy Distillation}},
  year = {2026},
  eprint = {2611.23034},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23034},
}
Figure