TL;DR
Researchers at NVIDIA have introduced Zone of Proximal Policy Optimization (ZPPO), a new method for training small AI models using large "teacher" models. Instead of forcing small models to mimic the teacher's internal math or injecting teacher answers directly into training gradients, ZPPO places the teacher’s guidance inside the prompts of hard questions, allowing the student to learn through its own reasoning.
What Problem Does This Paper Address?
Training small, efficient models (under 10B parameters) is essential for deploying AI on mobile phones or robots. Traditionally, this is done via Knowledge Distillation (KD), where a small "student" tries to imitate the exact output probabilities of a massive "teacher." However, for very small models, this is brittle; the student often memorizes specific answers rather than learning general rules, leading to poor performance on new types of problems.
An alternative is Reinforcement Learning (RL), where the model learns from its own successful attempts. But RL has a "blind spot": if a question is so hard that the student fails every single attempt, there is no success to learn from, and the question is simply discarded. Simply "plugging in" the teacher's correct answer into the RL math doesn't work either—it breaks the "on-policy" assumption of RL, causing the model's performance to drift and degrade.
Methods
ZPPO is inspired by Vygotsky’s "Zone of Proximal Development"—the idea that students learn best when tackling tasks they can’t quite do alone but can do with a little guidance.
The core innovation is keeping the teacher inside the prompt, not the gradient. When a student fails a question, ZPPO reformulates the prompt in two ways:
- Binary Candidate-included Question (BCQ): The prompt is updated to include two anonymized options: one correct response from the teacher and one incorrect response previously generated by the student. The student must then reason between them to find the right path.
- Negative Candidate-included Question (NCQ): The prompt aggregates several of the student's own failed attempts. This forces the student to look at its own errors collectively, helping it recognize and avoid shared failure patterns.
These reformulated questions are stored in a Prompt Replay Buffer. Hard questions are recirculated and practiced until the student "graduates" (reaches 50% accuracy on them) or the buffer runs out of space.
Datasets
The models were post-trained as Vision-Language Models (VLMs) using ZPPO-77K, a multimodal dataset of 77,000 image-question pairs curated from public sources like Vero and MMFineReason.
Evaluation was extensive, spanning a 31-benchmark suite:
- 16 VLM benchmarks (e.g., AI2D, ChartXiv, MMMUPro) to test core training alignment.
- 10 LLM benchmarks (e.g., MMLU, GSM8K, GPQA) to test generalization to text-only reasoning.
- 5 Video benchmarks (e.g., MVBench, Video-MME) to test generalization to temporal reasoning.
Key Insights and Ideas
- Teacher in Prompts > Teacher in Gradients: By providing the teacher's answer as a reference in the input rather than a target in the loss function, the student stays "on-policy." It is still generating its own reasoning, which makes the learning process more robust.
- The Power of Negative Examples: NCQ shows that a model can learn significantly just by being shown its own mistakes in a group. Seeing multiple ways it failed helps the model "triangulate" the correct logic.
- Super-additive Gains: The researchers found that combining the replay buffer with prompt reformulation was "super-additive"—the components together performed much better than the sum of their parts.
- Generalization vs. Memorization: Unlike standard distillation, which often hurts a model's ability to solve problems outside its training set, ZPPO improved performance across the board, even on video and text benchmarks the model wasn't specifically trained for.
Key Results
ZPPO was tested on the Qwen3.5 model family at four scales (0.8B, 2B, 4B, and 9B) using a 27B teacher.
- Smallest Models, Biggest Gains: The 0.8B student saw a massive +9.3 percentage point (pp) gain on VLM benchmarks compared to the base model.
- Outperforming Baselines: ZPPO consistently beat standard RL (GRPO) and both off-policy and on-policy distillation across all model sizes.
- Generalization: On benchmarks "beyond the training corpus" (LLM and Video), where distillation actually degraded performance, ZPPO provided significant improvements (e.g., +6.8 pp for the 0.8B model).
Limitations and Future Work
The authors identify a "structural ceiling": ZPPO relies on the teacher being able to solve the problem. If both the teacher and the student fail a question, there is no correct "candidate" to put in a BCQ prompt. In these cases, the system defaults to NCQ (learning from mistakes), which is less effective on its own.
Future work includes extending this "teacher-in-prompt" philosophy to multi-step agentic reasoning and exploring how to push a student's capabilities even in areas where the teacher might also struggle.
Summary updated