TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation

Lin Sun, Guangxiang Zhao, Xiaoqi Jian, Yuhan Wu, Weihong Lin, Yongfu Zhu, Change Jia, Linglin Zhang, Jinzhu Wu, Junfeng Ran, Saier Hu, Zihan Jiang, Junting Zhou, Wenrui Liu, Bin Cui, Tong Yang, Xiangzheng ZhangarXiv 2025 · DOI · arXiv

Introduction

Large-scale reasoning models like the 671B DeepSeek-R1 have set new benchmarks in complex problem-solving. However, distilling their capabilities into smaller, more efficient "student" models (e.g., 32B parameters) remains a significant challenge. Traditional distillation typically involves training the student on a massive, heterogeneous mixture of data. This "naive" approach often suffers from two primary issues: the Data Mixing Headache, where finding the optimal ratio of domain-specific data is a costly trial-and-error process, and Conflicting Gradients, where updates for one task (like coding) may degrade performance in another (like mathematics).

To address these challenges, we introduce TinyR1-32B-Preview, a model developed using a novel Branch-Merge Distillation strategy. This approach avoids multi-task interference by training specialized experts and then merging them into a unified, high-performance model.

The Branch-Merge Distillation Framework

The core of TinyR1's development is a two-phase process designed to maximize domain-specific expertise while maintaining a cohesive final architecture.

Phase 1: Branch (Specialization)

Instead of a single multi-task training run, the researchers created three distinct "expert" branches. Starting from the DeepSeek-R1-Distill-Qwen-32B base, they fine-tuned separate models on high-quality reasoning trajectories (Chain-of-Thought) for Math, Science, and Coding.

  • Math: 58k samples filtered from NuminaMath1.5.
  • Coding: 20k trajectories from the OpenThoughts dataset.
  • Science: 8.6k trajectories generated by DeepSeek-R1 based on seed examples.

This isolation allows each branch to reach its peak performance without the "seesaw effect" typically seen in multi-task learning.

Phase 2: Merge (Integration)

The specialized experts are then integrated using Arcee Fusion. Unlike standard weight averaging, this method utilizes a Kullback-Leibler (KL) divergence-based "Importance Score" to identify critical parameters within each expert. A dynamic selection process integrates these high-importance weights into the final TinyR1 model, ensuring that the specialized knowledge is preserved rather than diluted.

Overview of the Branch-Merge architecture and comparative performance across Math, Coding, and Science benchmarks.

Performance and Results

TinyR1-32B-Preview demonstrates significant improvements over its predecessor and even challenges much larger models. On the AIME 2024 (Math) benchmark, TinyR1 achieved a score of 78.1, nearly matching the 79.8 score of the full 671B DeepSeek-R1.

Key gains over the base DeepSeek-R1-Distill-Qwen-32B include:

  • Mathematics (AIME 2024): +5.5 points (78.1 vs 72.6)
  • Coding (LiveCodeBench): +4.4 points (61.6 vs 57.2)
  • Science (GPQA-Diamond): +2.9 points (65.0 vs 62.1)

Notably, TinyR1 outperforms the larger DeepSeek-R1-Distill-Llama-70B in both math and coding, despite having less than half the parameters.

Efficiency and the "Free Lunch" of Merging

One of the most striking findings is the computational efficiency of the Branch-Merge approach. Traditional multi-task retraining (the "Data Mixture" approach) required approximately 740 GPU hours. In contrast, the merging phase of TinyR1 took only 4 GPU hours—a 99% reduction in compute. Remarkably, the merged model not only saved time but also yielded superior performance compared to the data mixture baseline, effectively solving the gradient conflict problem.

Ablation study comparing specialized experts, data mixture training, and different merging sequences, highlighting the 99% reduction in compute time.

Conclusion and Future Directions

TinyR1-32B-Preview proves that specialized distillation followed by intelligent merging is a highly effective alternative to standard multi-task distillation. By mitigating the "seesaw problem" of conflicting gradients, the Branch-Merge strategy allows smaller models to punch well above their weight class.

Future work will focus on exploring different model backbones, such as Qwen-Instruct, and scaling the Branch-Merge approach to various model sizes. While the current model's long Chain-of-Thought reasoning increases inference costs, the gains in accuracy and the massive reduction in training compute make it a compelling path forward for efficient LLM development.

Summary updated

BibTeX
@misc{sun2025tinyr,
  author = {Lin Sun and Guangxiang Zhao and Xiaoqi Jian and Yuhan Wu and Weihong Lin and Yongfu Zhu and Change Jia and Linglin Zhang and Jinzhu Wu and Junfeng Ran and Saier Hu and Zihan Jiang and Junting Zhou and Wenrui Liu and Bin Cui and Tong Yang and Xiangzheng Zhang},
  title = {{TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation}},
  year = {2025},
  eprint = {2503.04872},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.04872},
}
Figure