From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL

Yitong Qiao, Tiantian He, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu, Zhixuan ChuarXiv 2026 · arXiv

TL;DR

This paper introduces On-Policy Warmup (OPW), a teacher-guided initialization strategy for training language model agents using reinforcement learning with verifiable rewards (RLVR). By supervising a student model on its own interaction trajectories before transitioning to standard RL, OPW overcomes the "cold-start" problem of sparse rewards, leading to a ~1.5x training speedup and significantly higher final task performance.

What Problem Does This Paper Address?

Training autonomous agents (like those that use tools or navigate environments) often relies on Reinforcement Learning with Verifiable Rewards (RLVR). In this setup, the model only gets a reward if it successfully completes a complex, multi-step task (e.g., "clean a pan and put it on the counter").

The challenge is the "cold-start" problem: early in training, a model might fail 99% of the time. Because the rewards are sparse and binary, the RL algorithm receives almost no signal to distinguish "better" failures from "worse" ones. While standard imitation learning (Supervised Fine-Tuning or SFT) can help, it suffers from distribution mismatch: the student is trained on perfect teacher trajectories, but if it makes a single mistake during its own rollout, it enters a state the teacher never visited and doesn't know how to recover.

Methods

The authors propose a two-stage pipeline:

  1. Stage I: On-Policy Warmup (OPW): The student model interacts with the environment to generate trajectories. A fixed, larger "teacher" model (e.g., Qwen3-32B) provides supervision for every step the student took, even if the student's actions were imperfect. This uses on-policy reverse-KL distillation, which forces the student to match the teacher's distribution specifically on states the student is likely to visit.
  2. Stage II: Verifiable-Based RL: Once the student has reached a baseline level of competence, teacher supervision is removed. The model continues training using Group-Relative Policy Optimization (GRPO), guided only by the environment's verifiable success/failure rewards.

The key innovation is the "on-policy" nature of the warmup: the teacher corrects the student's actual mistakes and recovery attempts, rather than just showing the student a "perfect" path it can't replicate.

Datasets

The researchers evaluated their approach on two complex, multi-step interactive benchmarks:

  • ALFWorld: A synthetic environment where agents must complete household tasks (cleaning, heating, placing objects) using text-based commands.
  • ScienceWorld: A more challenging benchmark requiring agents to conduct scientific experiments (e.g., testing conductivity, changing the state of matter) in a simulated laboratory.

Key Insights and Ideas

  • The On-Policy Acceleration Phenomenon: The authors observed that starting with on-policy distillation doesn't just provide a better starting point; it fundamentally changes the learning curve, allowing the model to reach high performance much earlier and sustain a higher final ceiling than baselines.
  • Trajectory-Level Distribution Matching: The paper provides a theoretical proof that minimizing on-policy distillation loss directly bounds the gap between the teacher's and student's success probabilities. This ensures the student "discovers" successful trajectories early enough for RL to take over.
  • Reward Discovery & Informative Groups: For group-relative RL methods like GRPO, training only works if a batch (group) contains both successes and failures. OPW pushes the student's success probability into the "sweet spot" (around 50%) where the likelihood of sampling these "reward-informative groups" is maximized.
  • Error Recovery: Unlike offline SFT, OPW teaches the model how to handle "recovery states." If the model issues an invalid command, the teacher provides the correct next step from that specific failure point.

Key Results

  • Efficiency: As shown in Figure 1, OPW achieves a ~1.51x speedup in training GPU-hours to reach the same performance level as direct GRPO.
  • Performance: On "Unseen" tasks in ALFWorld, the OPW + GRPO pipeline reached 96.7% success, compared to 86.6% for direct GRPO and only 53.5% for direct on-policy distillation (OPD).
  • Robustness: The models trained with OPW showed much higher repeated task success (pass^k). For Qwen3-4B, the probability of succeeding in 8 out of 8 rollouts (pass^8) was significantly higher than baselines, indicating a more stable and reliable policy.
OPW accelerates training and improves robustness (pass^k) compared to standard GRPO and SFT baselines.
  • Diversity of Success: OPW helps the model discover a wider variety of successful action sequences early in the RL phase, which prevents the model from collapsing into a single, fragile path to success.
OPW leads to a higher count of distinct successful action sequences early in training.

Limitations and Future Work

  • Teacher Dependence: The method requires access to a competent teacher model, which may be computationally expensive or unavailable for extremely niche domains.
  • Warmup Duration: The authors noted that the optimal duration for the warmup phase is currently a hyperparameter. If the warmup is too short, the model doesn't discover enough rewards; if it's too long, it wastes compute that could be better spent on environment-specific RL.
  • Transition Criteria: Future work could involve developing adaptive criteria to automatically trigger the transition from teacher-guided warmup to reward-based RL based on the model's internal success metrics.

Summary updated

Abstract

Reinforcement learning with a verifiable reward (RLVR) offers a scalable approach to training language-model agents, yet sparse outcome rewards can leave early training with little signal for policy improvement. We identify an On-Policy Acceleration Phenomenon: in our main comparisons, RLVR initialized with on-policy distillation reaches high performance earlier in training and achieves both higher average performance during subsequent RLVR and higher final performance than the alternative baselines. Motivated by this observation, we study On-Policy Warmup (OPW), a teacher-guided stage in which the student trains with teacher supervision on its own interaction trajectories before transitioning to RLVR. Unlike imitation on fixed teacher-generated trajectories, OPW targets states induced by the student's own decisions, including imperfect actions and recovery situations. We provide a theoretical explanation by connecting on-policy reverse-KL distillation to trajectory-level distribution matching. Under a competent teacher and sufficiently small population distillation loss, this connection yields a lower bound on initial verifier success and a corresponding bound on reward-discovery complexity. For group-relative RLVR, we further characterize when increased success probability produces more reward-informative groups. Together, our findings support on-policy distillation as an effective warmup for agentic RLVR and identify initial reward discovery as a mechanism that can contribute to the observed acceleration.
BibTeX
@misc{qiao2026from,
  author = {Yitong Qiao and Tiantian He and Lei Liu and Yue Shen and Jian Wang and Jinjie Gu and Zhixuan Chu},
  title = {{From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL}},
  year = {2026},
  eprint = {2609.39436},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.39436},
}
Figure