PIVOT: Progressive Intervention via Online Teacher Correction

Zhihao Yang, Boyuan Zheng, Puhui Tan, Yukun Chen, Kuan Zhang, Longze Chen, Jiaming Li, Renke Shan, Lu Wang, Xianpei HanarXiv 2026 · arXiv

TL;DR

PIVOT overcomes the "cold start" problem in reinforcement learning for reasoning by having a teacher model intervene mid-rollout to correct student errors. By combining these corrected trajectories with dense token-level supervision, PIVOT nearly doubles performance on hard logic games and significantly outperforms standard RL and distillation baselines on complex mathematical tasks.

What Problem Does This Paper Address?

Reinforcement Learning from Verifiable Rewards (RLVR)—the paradigm behind models like DeepSeek-R1—has revolutionized how we elicit reasoning in Large Language Models (LLMs). However, it hits a wall on "hard" problems. If a student model is so weak that it fails every single attempt during training, the verifier only returns zero rewards. With no successful examples to learn from, the model receives no learning signal and cannot improve.

While we can use stronger "teacher" models to provide guidance, existing methods usually pick one of two flawed paths:

  1. Trajectory-level intervention: The teacher provides a full correct path. The problem? These paths might be too different from how the student "thinks," making them hard to imitate.
  2. Token-level supervision: The teacher provides feedback on the student's own attempts. The problem? If the student makes an early error, the teacher often gets "conflicted" between trying to fix the mistake and trying to follow the student's flawed logic, providing a noisy and confusing signal.

Methods

The authors introduce PIVOT (Progressive Intervention via Online Teacher Correction), which treats teacher guidance as a surgical intervention rather than a blanket replacement.

The PIVOT workflow for a single training query looks like this:

  1. Student Prefix: The student model starts solving the problem for a short duration (e.g., the first 512 tokens).
  2. Teacher Intervention: A stronger teacher model takes over, generating a "correction segment" that builds on the student's work but steers it toward a correct solution.
  3. Student Continuation: The student model takes the reins back and finishes the trajectory based on the teacher's correction.
  4. Hybrid Optimization: The model is trained using a combination of the standard GRPO (Group Relative Policy Optimization) loss on the whole trajectory and a Teacher-Student KL Loss applied specifically to the tokens in the teacher's correction segment.
Figure 1. Overview of PIVOT. For each query, the policy model samples a group of rollouts, including teacher-assisted trajectories. In a teacher-assisted rollout, the student first generates a prefix, the teacher provides a correction segment, and the student then continues generation. The completed trajectory is optimized by the verifier-based GRPO objective, while the teacher-generated segment provides dense token-level supervision through a teacher-student KL loss.

Datasets

The researchers evaluated PIVOT on two main fronts:

  • Hard Logic Games: Subsets of SynLogic (Survo, Minesweeper, and Kakurasu) where the base models initially have near-zero accuracy.
  • Mathematical Reasoning: A pool of 30k challenging problems from Polaris-53K and Skywork-OR1-RL-Data.
  • Out-of-Distribution (OOD) Math: Evaluation on AIME 2024/2025 and MATH500 to test generalization.

Key Insights and Ideas

  • Middle Intervention is the "Sweet Spot": The authors found that intervening after the student makes a short attempt is far more effective than providing a teacher-generated prefix from the start. This "student-conditioned" guidance is more learnable because the teacher is helping the student recover from its own specific reasoning states.
  • Resolving Teacher Conflict: In standard distillation (like KDRL), if a student makes an error, the teacher's probability distribution often splits between "fixing the error" and "continuing the wrong path." PIVOT forces the teacher to actually perform the correction, creating a clear, unimodal target for the student to learn from.
Figure 3. Dense supervision under KDRL and PIVOT on a Survo rollout; insets show the teacher’s top-2 next-token probabilities at supervised positions. Left: on the student’s failed trajectory, the teacher is split between fixing the error (“Wait”) and continuing along it (“Let”, “So”), giving conflicting targets. Right: the teacher corrects the error itself and continues from there, so its targets are unimodal whether it backtracks (“Wait”) or proceeds (“So”), and the trajectory reaches the reward.
  • KL Direction Matters: The choice of KL divergence objective depends on the student's initial skill. Reverse KL (which is "mode-seeking") works best when the student is failing completely, as it helps the model lock onto a single viable strategy. Forward KL (which is "mode-covering") is slightly better when the student is already competent and needs to learn a broader range of solutions.
  • Quality over Quantity: When working with a fixed "teacher token budget," it is better to have a few long, sustained teacher interventions than many short, choppy ones. Longer segments allow the teacher to fully develop the reasoning required for a recovery.

Key Results

PIVOT demonstrated massive gains over state-of-the-art baselines:

  • Logic Games: PIVOT (Reverse KL) nearly doubled the mean avg@8 score from 19.5% (the best baseline, KDRL) to 35.6%. On the Survo game specifically, performance skyrocketed from a base of 0.05% to 72.9%.
  • Mathematical Reasoning: On the in-distribution Math-ID set, PIVOT reached 23.05% accuracy, significantly higher than the base model's 5.04% and outperforming other teacher-based methods like POPE (22.09%).
  • Generalization: PIVOT consistently improved scores on OOD benchmarks like AIME, whereas some baselines actually caused performance to regress compared to the base model.

Limitations and Future Work

  • Task Scope: The study focused heavily on logic and math. The authors acknowledge that the effectiveness of PIVOT on code generation or open-ended creative reasoning remains to be seen.
  • Static Intervention: Currently, PIVOT uses a fixed intervention point (e.g., always at 512 tokens). A more advanced version might use "adaptive intervention," where the teacher only steps in when it detects the student has actually made a logical error.
  • Evolving Needs: As the student improves during training, the type of guidance it needs likely changes. Future work could explore how to dynamically shift the teacher's role from "corrector" to "refiner."

Summary updated

Abstract

Reinforcement learning from verifiable rewards (RLVR) is limited by sparse outcome-level feedback on hard reasoning tasks: when all rollouts in a group fail, rewards provide no learning signal. Stronger teachers can provide additional guidance, but this raises a central question: how can such signals be incorporated into RLVR to support successful rollouts while remaining learnable for the student? We distinguish two complementary mechanisms: trajectory-level intervention, which improves access to reward-bearing trajectories, and token-level supervision, which provides dense targets to guide student learning. Our controlled experiments show that either mechanism alone can remain insufficient, motivating their combination. We therefore introduce PIVOT, which inserts teacher-generated corrections into student rollouts and applies token-level KL supervision to these segments alongside the RLVR objective. Beyond their combination, we systematically analyze where to intervene and how to supervise, yielding a recipe for effective guidance. Across hard logic games and mathematical reasoning benchmarks, PIVOT consistently outperforms existing methods, nearly doubling mean avg@8 on logic games from 19.5% for the best-performing baseline to 35.6%.
BibTeX
@misc{yang2026pivot,
  author = {Zhihao Yang and Boyuan Zheng and Puhui Tan and Yukun Chen and Kuan Zhang and Longze Chen and Jiaming Li and Renke Shan and Lu Wang and Xianpei Han},
  title = {{PIVOT: Progressive Intervention via Online Teacher Correction}},
  year = {2026},
  eprint = {2610.40576},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40576},
}
Figure