TL;DR
This paper introduces On-Policy Warmup (OPW), a teacher-guided initialization strategy for training language model agents using reinforcement learning with verifiable rewards (RLVR). By supervising a student model on its own interaction trajectories before transitioning to standard RL, OPW overcomes the "cold-start" problem of sparse rewards, leading to a ~1.5x training speedup and significantly higher final task performance.
What Problem Does This Paper Address?
Training autonomous agents (like those that use tools or navigate environments) often relies on Reinforcement Learning with Verifiable Rewards (RLVR). In this setup, the model only gets a reward if it successfully completes a complex, multi-step task (e.g., "clean a pan and put it on the counter").
The challenge is the "cold-start" problem: early in training, a model might fail 99% of the time. Because the rewards are sparse and binary, the RL algorithm receives almost no signal to distinguish "better" failures from "worse" ones. While standard imitation learning (Supervised Fine-Tuning or SFT) can help, it suffers from distribution mismatch: the student is trained on perfect teacher trajectories, but if it makes a single mistake during its own rollout, it enters a state the teacher never visited and doesn't know how to recover.
Methods
The authors propose a two-stage pipeline:
- Stage I: On-Policy Warmup (OPW): The student model interacts with the environment to generate trajectories. A fixed, larger "teacher" model (e.g., Qwen3-32B) provides supervision for every step the student took, even if the student's actions were imperfect. This uses on-policy reverse-KL distillation, which forces the student to match the teacher's distribution specifically on states the student is likely to visit.
- Stage II: Verifiable-Based RL: Once the student has reached a baseline level of competence, teacher supervision is removed. The model continues training using Group-Relative Policy Optimization (GRPO), guided only by the environment's verifiable success/failure rewards.
The key innovation is the "on-policy" nature of the warmup: the teacher corrects the student's actual mistakes and recovery attempts, rather than just showing the student a "perfect" path it can't replicate.
Datasets
The researchers evaluated their approach on two complex, multi-step interactive benchmarks:
- ALFWorld: A synthetic environment where agents must complete household tasks (cleaning, heating, placing objects) using text-based commands.
- ScienceWorld: A more challenging benchmark requiring agents to conduct scientific experiments (e.g., testing conductivity, changing the state of matter) in a simulated laboratory.
Key Insights and Ideas
- The On-Policy Acceleration Phenomenon: The authors observed that starting with on-policy distillation doesn't just provide a better starting point; it fundamentally changes the learning curve, allowing the model to reach high performance much earlier and sustain a higher final ceiling than baselines.
- Trajectory-Level Distribution Matching: The paper provides a theoretical proof that minimizing on-policy distillation loss directly bounds the gap between the teacher's and student's success probabilities. This ensures the student "discovers" successful trajectories early enough for RL to take over.
- Reward Discovery & Informative Groups: For group-relative RL methods like GRPO, training only works if a batch (group) contains both successes and failures. OPW pushes the student's success probability into the "sweet spot" (around 50%) where the likelihood of sampling these "reward-informative groups" is maximized.
- Error Recovery: Unlike offline SFT, OPW teaches the model how to handle "recovery states." If the model issues an invalid command, the teacher provides the correct next step from that specific failure point.
Key Results
- Efficiency: As shown in Figure 1, OPW achieves a ~1.51x speedup in training GPU-hours to reach the same performance level as direct GRPO.
- Performance: On "Unseen" tasks in ALFWorld, the OPW + GRPO pipeline reached 96.7% success, compared to 86.6% for direct GRPO and only 53.5% for direct on-policy distillation (OPD).
- Robustness: The models trained with OPW showed much higher repeated task success (pass^k). For Qwen3-4B, the probability of succeeding in 8 out of 8 rollouts (pass^8) was significantly higher than baselines, indicating a more stable and reliable policy.
- Diversity of Success: OPW helps the model discover a wider variety of successful action sequences early in the RL phase, which prevents the model from collapsing into a single, fragile path to success.
Limitations and Future Work
- Teacher Dependence: The method requires access to a competent teacher model, which may be computationally expensive or unavailable for extremely niche domains.
- Warmup Duration: The authors noted that the optimal duration for the warmup phase is currently a hyperparameter. If the warmup is too short, the model doesn't discover enough rewards; if it's too long, it wastes compute that could be better spent on environment-specific RL.
- Transition Criteria: Future work could involve developing adaptive criteria to automatically trigger the transition from teacher-guided warmup to reward-based RL based on the model's internal success metrics.
Summary updated