Introduction
Large-scale reasoning models like the 671B DeepSeek-R1 have set new benchmarks in complex problem-solving. However, distilling their capabilities into smaller, more efficient "student" models (e.g., 32B parameters) remains a significant challenge. Traditional distillation typically involves training the student on a massive, heterogeneous mixture of data. This "naive" approach often suffers from two primary issues: the Data Mixing Headache, where finding the optimal ratio of domain-specific data is a costly trial-and-error process, and Conflicting Gradients, where updates for one task (like coding) may degrade performance in another (like mathematics).
To address these challenges, we introduce TinyR1-32B-Preview, a model developed using a novel Branch-Merge Distillation strategy. This approach avoids multi-task interference by training specialized experts and then merging them into a unified, high-performance model.
The Branch-Merge Distillation Framework
The core of TinyR1's development is a two-phase process designed to maximize domain-specific expertise while maintaining a cohesive final architecture.
Phase 1: Branch (Specialization)
Instead of a single multi-task training run, the researchers created three distinct "expert" branches. Starting from the DeepSeek-R1-Distill-Qwen-32B base, they fine-tuned separate models on high-quality reasoning trajectories (Chain-of-Thought) for Math, Science, and Coding.
- Math: 58k samples filtered from NuminaMath1.5.
- Coding: 20k trajectories from the OpenThoughts dataset.
- Science: 8.6k trajectories generated by DeepSeek-R1 based on seed examples.
This isolation allows each branch to reach its peak performance without the "seesaw effect" typically seen in multi-task learning.
Phase 2: Merge (Integration)
The specialized experts are then integrated using Arcee Fusion. Unlike standard weight averaging, this method utilizes a Kullback-Leibler (KL) divergence-based "Importance Score" to identify critical parameters within each expert. A dynamic selection process integrates these high-importance weights into the final TinyR1 model, ensuring that the specialized knowledge is preserved rather than diluted.
Performance and Results
TinyR1-32B-Preview demonstrates significant improvements over its predecessor and even challenges much larger models. On the AIME 2024 (Math) benchmark, TinyR1 achieved a score of 78.1, nearly matching the 79.8 score of the full 671B DeepSeek-R1.
Key gains over the base DeepSeek-R1-Distill-Qwen-32B include:
- Mathematics (AIME 2024): +5.5 points (78.1 vs 72.6)
- Coding (LiveCodeBench): +4.4 points (61.6 vs 57.2)
- Science (GPQA-Diamond): +2.9 points (65.0 vs 62.1)
Notably, TinyR1 outperforms the larger DeepSeek-R1-Distill-Llama-70B in both math and coding, despite having less than half the parameters.
Efficiency and the "Free Lunch" of Merging
One of the most striking findings is the computational efficiency of the Branch-Merge approach. Traditional multi-task retraining (the "Data Mixture" approach) required approximately 740 GPU hours. In contrast, the merging phase of TinyR1 took only 4 GPU hours—a 99% reduction in compute. Remarkably, the merged model not only saved time but also yielded superior performance compared to the data mixture baseline, effectively solving the gradient conflict problem.
Conclusion and Future Directions
TinyR1-32B-Preview proves that specialized distillation followed by intelligent merging is a highly effective alternative to standard multi-task distillation. By mitigating the "seesaw problem" of conflicting gradients, the Branch-Merge strategy allows smaller models to punch well above their weight class.
Future work will focus on exploring different model backbones, such as Qwen-Instruct, and scaling the Branch-Merge approach to various model sizes. While the current model's long Chain-of-Thought reasoning increases inference costs, the gains in accuracy and the massive reduction in training compute make it a compelling path forward for efficient LLM development.
Summary updated