Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

Byung-Kwan Lee, Ximing Lu, Shizhe Diao, Minki Kang, Saurav Muralidharan, Karan Sapra, Andrew Tao, Pavlo Molchanov, Yejin Choi, Yu-Chiang Frank Wang, Ryo HachiumaarXiv 2026 · arXiv

TL;DR

Researchers at NVIDIA have introduced Zone of Proximal Policy Optimization (ZPPO), a new method for training small AI models using large "teacher" models. Instead of forcing small models to mimic the teacher's internal math or injecting teacher answers directly into training gradients, ZPPO places the teacher’s guidance inside the prompts of hard questions, allowing the student to learn through its own reasoning.

What Problem Does This Paper Address?

Training small, efficient models (under 10B parameters) is essential for deploying AI on mobile phones or robots. Traditionally, this is done via Knowledge Distillation (KD), where a small "student" tries to imitate the exact output probabilities of a massive "teacher." However, for very small models, this is brittle; the student often memorizes specific answers rather than learning general rules, leading to poor performance on new types of problems.

An alternative is Reinforcement Learning (RL), where the model learns from its own successful attempts. But RL has a "blind spot": if a question is so hard that the student fails every single attempt, there is no success to learn from, and the question is simply discarded. Simply "plugging in" the teacher's correct answer into the RL math doesn't work either—it breaks the "on-policy" assumption of RL, causing the model's performance to drift and degrade.

Methods

ZPPO is inspired by Vygotsky’s "Zone of Proximal Development"—the idea that students learn best when tackling tasks they can’t quite do alone but can do with a little guidance.

Failure modes of traditional methods vs. the ZPPO approach.

The core innovation is keeping the teacher inside the prompt, not the gradient. When a student fails a question, ZPPO reformulates the prompt in two ways:

  1. Binary Candidate-included Question (BCQ): The prompt is updated to include two anonymized options: one correct response from the teacher and one incorrect response previously generated by the student. The student must then reason between them to find the right path.
  2. Negative Candidate-included Question (NCQ): The prompt aggregates several of the student's own failed attempts. This forces the student to look at its own errors collectively, helping it recognize and avoid shared failure patterns.
Overview of the ZPPO training loop and prompt reformulation.

These reformulated questions are stored in a Prompt Replay Buffer. Hard questions are recirculated and practiced until the student "graduates" (reaches 50% accuracy on them) or the buffer runs out of space.

Datasets

The models were post-trained as Vision-Language Models (VLMs) using ZPPO-77K, a multimodal dataset of 77,000 image-question pairs curated from public sources like Vero and MMFineReason.

Evaluation was extensive, spanning a 31-benchmark suite:

  • 16 VLM benchmarks (e.g., AI2D, ChartXiv, MMMUPro) to test core training alignment.
  • 10 LLM benchmarks (e.g., MMLU, GSM8K, GPQA) to test generalization to text-only reasoning.
  • 5 Video benchmarks (e.g., MVBench, Video-MME) to test generalization to temporal reasoning.

Key Insights and Ideas

  • Teacher in Prompts > Teacher in Gradients: By providing the teacher's answer as a reference in the input rather than a target in the loss function, the student stays "on-policy." It is still generating its own reasoning, which makes the learning process more robust.
  • The Power of Negative Examples: NCQ shows that a model can learn significantly just by being shown its own mistakes in a group. Seeing multiple ways it failed helps the model "triangulate" the correct logic.
  • Super-additive Gains: The researchers found that combining the replay buffer with prompt reformulation was "super-additive"—the components together performed much better than the sum of their parts.
  • Generalization vs. Memorization: Unlike standard distillation, which often hurts a model's ability to solve problems outside its training set, ZPPO improved performance across the board, even on video and text benchmarks the model wasn't specifically trained for.

Key Results

ZPPO was tested on the Qwen3.5 model family at four scales (0.8B, 2B, 4B, and 9B) using a 27B teacher.

  • Smallest Models, Biggest Gains: The 0.8B student saw a massive +9.3 percentage point (pp) gain on VLM benchmarks compared to the base model.
  • Outperforming Baselines: ZPPO consistently beat standard RL (GRPO) and both off-policy and on-policy distillation across all model sizes.
  • Generalization: On benchmarks "beyond the training corpus" (LLM and Video), where distillation actually degraded performance, ZPPO provided significant improvements (e.g., +6.8 pp for the 0.8B model).
Macro-average gains of ZPPO across LLM, VLM, and Video benchmarks.

Limitations and Future Work

The authors identify a "structural ceiling": ZPPO relies on the teacher being able to solve the problem. If both the teacher and the student fail a question, there is no correct "candidate" to put in a BCQ prompt. In these cases, the system defaults to NCQ (learning from mistakes), which is less effective on its own.

Future work includes extending this "teacher-in-prompt" philosophy to multi-step agentic reasoning and exploring how to push a student's capabilities even in areas where the teacher might also struggle.

Summary updated

Abstract

Knowledge distillation transfers a teacher's competence to a small student but is brittle in the small-student regime: forcing the student to imitate logits from a much larger teacher concentrates it on the teacher's sharpest modes, hurting generalization on benchmark families beyond the training corpus. Reinforcement learning (RL) avoids logit imitation by training on the student's own rollouts. However, on questions where every rollout fails, yielding zero advantage and being silently discarded, injecting a stronger teacher's response into the policy gradient breaks the on-policy assumption and induces drift. We introduce Zone of Proximal Policy Optimization (ZPPO), inspired by Vygotsky's zone of proximal development, which keeps the teacher inside the prompt rather than the policy gradient. On hard questions, ZPPO constructs two reformulated prompts. A Binary Candidate-included Question (BCQ) pairs one correct teacher response with one incorrect student response as anonymized candidates the student must reason between. A Negative Candidate-included Question (NCQ) aggregates the student's wrong rollouts into a single prompt to surface their shared failure modes. A prompt replay buffer recirculates each hard question until it either graduates (the student's mean rollout accuracy on it reaches half or more) or is FIFO-evicted under finite capacity, amplifying BCQ and NCQ inside the student's current zone of proximal development. On the Qwen3.5 family at four student scales (0.8B-9B) with a 27B teacher, post-trained as vision-language models and evaluated on a 31-benchmark suite (16 VLM, 10 LLM, 5 Video), ZPPO outperforms off/on-policy distillation and GRPO, with the largest gains at the smallest scale.
BibTeX
@misc{lee2026zone,
  author = {Byung-Kwan Lee and Ximing Lu and Shizhe Diao and Minki Kang and Saurav Muralidharan and Karan Sapra and Andrew Tao and Pavlo Molchanov and Yejin Choi and Yu-Chiang Frank Wang and Ryo Hachiuma},
  title = {{Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients}},
  year = {2026},
  eprint = {2606.18216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.18216},
}
Figure