KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning

Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, Fei MiarXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces KDRL, a novel unified framework for post-training Large Language Models (LLMs) to improve their reasoning abilities. By jointly optimizing knowledge distillation (KD) and reinforcement learning (RL) through a single objective, KDRL achieves better performance and efficiency than existing separate approaches.

What Problem Does This Paper Address?

Large Language Models (LLMs) often struggle with complex reasoning tasks. While two main post-training strategies exist – Reinforcement Learning (RL) and Knowledge Distillation (KD) – they each have limitations. RL can develop sophisticated reasoning but is sample-inefficient, especially when the initial model is weak. KD is efficient but can generalize poorly to unseen data. Current methods often apply these sequentially, which might not be optimal. This paper aims to bridge this gap by creating a unified framework that leverages the strengths of both KD and RL simultaneously for more effective and efficient reasoning enhancement.

Methods

The core of KDRL is a unified objective that combines Reinforcement Learning (specifically GRPO, a policy gradient method) with Knowledge Distillation using Reverse KL divergence (KD-RKL). Instead of separate training stages, KDRL optimizes a single objective function. This involves:

  • Unified Objective: A policy gradient objective that simultaneously minimizes the Reverse KL divergence (RKL) between the student and teacher models and maximizes rule-based rewards from a verifier.
  • KD-RKL: Uses on-policy distillation by minimizing RKL between the student and teacher distributions on self-generated data, mitigating exposure bias common in off-policy KD.
  • KL Approximations: Explores different methods (k2, k3, Top-K) to estimate the KL divergence, with the unbiased k2 estimator showing superior performance.
  • KL Coefficient (β): Investigates the trade-off between KD and RL by tuning a coefficient, finding that annealing this coefficient (gradually reducing KD's influence) leads to better performance.
  • Reward-Guided KD: Introduces masking strategies to suppress KD for already well-performing responses, allowing distillation to focus on harder examples.

Datasets

The paper uses several mathematical reasoning datasets for training and evaluation:

  • Training Data: Skywork-OR1 dataset (a large collection of math problems), AIME, AMC, and Omni-Math.
  • Evaluation Datasets: MATH500, MinervaMath, OlympiadBench, AIME24, AIME25, and AMC23.

Key Insights and Ideas

  • Synergy of KD and RL: Combining KD and RL in a unified framework is more effective than sequential application. The unified objective allows for a more balanced learning process.
  • On-Policy KD is Crucial: Using Reverse KL (RKL) for KD, which operates on self-generated data, is vital for avoiding exposure bias and achieving better performance compared to standard off-policy KD.
  • Annealing the KL Coefficient: A dynamic annealing schedule for the KL coefficient (β) is more beneficial than a fixed one. It allows for strong teacher guidance early on and a smoother transition to reward-driven optimization, leading to better performance and efficiency.
  • Focusing KD on "Hard" Examples: Reward-guided masking, particularly response-level masking, improves efficiency by directing distillation efforts towards responses that need more improvement, rather than over-regularizing already good ones.

Key Results

  • Performance Improvement: KDRL (specifically KDRL-Annealing) consistently outperforms baselines like GRPO (pure RL) and KD-RKL (on-policy KD) across multiple mathematical reasoning benchmarks. For instance, KDRL-Annealing achieved the highest overall accuracy, improving over SFT, GRPO, and KD-RKL by 4.7%, 2.6%, and 1.1%, respectively.
  • Efficiency Gains: KDRL achieves a favorable trade-off between performance and efficiency. It produces moderately long responses, balancing generation efficiency and performance, unlike KD-RKL which can lead to significantly longer outputs.
  • KL Approximation Matters: The unbiased k2 KL estimator in KDRL consistently outperforms other approximations like k3 and Top-K, which can lead to unstable training.
  • Teacher Model Impact: The choice of teacher model significantly impacts performance, with a stronger teacher leading to better results.

Limitations and Future Work

  • Teacher Model Dependency: The performance of KDRL is still dependent on the quality of the teacher model. Using a weaker teacher model resulted in less significant gains compared to using a stronger one.
  • Computational Cost: While KDRL improves efficiency, RL-based methods inherently require more computational resources than pure KD.
  • Future Work: The paper suggests exploring further improvements through more sophisticated KL approximations, advanced annealing schedules, and more complex reward functions. The authors also highlight the potential of their unified framework for other reasoning tasks beyond mathematics.

Summary updated

Abstract

Recent advances in large language model (LLM) post-training have leveraged two distinct paradigms to enhance reasoning capabilities: reinforcement learning (RL) and knowledge distillation (KD). While RL enables the emergence of complex reasoning behaviors, it often suffers from low sample efficiency when the initial policy struggles to explore high-reward trajectories. Conversely, KD improves learning efficiency via mimicking the teacher model but tends to generalize poorly to out-of-domain scenarios. In this work, we present \textbf{KDRL}, a \textit{unified post-training framework} that jointly optimizes a reasoning model through teacher supervision (KD) and self-exploration (RL). Specifically, KDRL leverages policy gradient optimization to simultaneously minimize the reverse Kullback-Leibler divergence (RKL) between the student and teacher distributions while maximizing the expected rule-based rewards. We first formulate a unified objective that integrates GRPO and KD, and systematically explore how different KL approximations, KL coefficients, and reward-guided KD strategies affect the overall post-training dynamics and performance. Empirical results on multiple reasoning benchmarks demonstrate that KDRL outperforms GRPO and various KD baselines while achieving a favorable balance between performance and reasoning token efficiency. These findings indicate that integrating KD and RL serves as an effective and efficient strategy to train reasoning LLMs.
BibTeX
@misc{xu2025kdrl,
  author = {Hongling Xu and Qi Zhu and Heyuan Deng and Jinpeng Li and Lu Hou and Yasheng Wang and Lifeng Shang and Ruifeng Xu and Fei Mi},
  title = {{KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning}},
  year = {2025},
  eprint = {2506.02208},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.02208},
}
Figure