TL;DR
On-policy distillation (OPD) is the standard for transferring model behavior, but it fails miserably at injecting new knowledge because the student can't learn what it doesn't already "suspect." Adaptive-Policy Distillation (APD) solves this by using a per-token gate that selectively switches to teacher-led sampling and forward-KL loss only at the specific positions where the student is most clueless.
What Problem Does This Paper Address?
Large Language Models (LLMs) are often used to answer questions about specific, long documents (technical manuals, legal texts, etc.). While putting these documents in the prompt (RAG) works well, it's computationally expensive and eats up context window capacity. Context distillation aims to "bake" that document knowledge directly into the model's weights, enabling "closed-book" inference.
The current state-of-the-art for behavior transfer is On-Policy Distillation (OPD), where the student generates its own answers and is supervised by a teacher who has access to the document. However, the authors discovered a "surprising reversal": for knowledge injection, OPD is actually worse than simpler off-policy methods.
The problem is Information Asymmetry. Because the student doesn't have the document, it assigns near-zero probability to document-specific tokens. This triggers two structural failures:
- Vanishing Gradients: The standard Reverse-KL loss used in OPD has a gradient that scales with the student's own probability. If the student thinks a token is impossible, the gradient vanishes, and it never learns to raise that probability.
- The Coverage Wall: In on-policy rollouts, if a student misses the first token of a fact, it wanders off-track and never even sees the subsequent tokens it needs to learn.
Methods
The authors propose Adaptive-Policy Distillation (APD). Instead of a fixed training recipe, APD uses a per-token divergence gate ($g_t$) based on the KL divergence between the teacher and student at each step.
As shown in Figure 1, the failure of standard methods is tied to both the KL direction and the rollout source. APD adaptively intervenes on both:
- Selective Sampling: When the gate opens (high divergence), the next token is drawn from a mixture of the student and teacher. This keeps the rollout "on track" to expose the student to the correct knowledge.
- Selective Loss: The gate blends the loss between Reverse-KL (good for maintaining general behavior) and Forward-KL (which provides a strong signal even for low-probability tokens).
As the student learns the document, the divergence drops, the gate naturally closes, and the process returns to standard on-policy distillation.
Datasets
The researchers evaluated APD using:
- A Synthetic Testbed: A controlled environment using random key-code mappings (e.g., "The code for X is 1234"). This allowed them to independently vary "unpredictability" (alphabet size) and "length" to isolate exactly where OPD fails.
- Real-World Benchmarks:
- QASPER: Information-seeking QA over full NLP research papers.
- BioASQ: Biomedical factoid QA over full PMC articles. They used Llama-3.1-8B-Instruct and Qwen3-8B as the base models for these experiments.
Key Insights and Ideas
- The "Zero-Forcing" Trap: The paper mathematically demonstrates that Reverse-KL is "zero-forcing." It's great for making a model concise, but it's the enemy of knowledge injection because it provides no "first-order signal" to raise the probability of a token the student currently considers unlikely.
- Knowledge is Sparse: Most of a document is "filler" or common language the student already knows. The "hard" knowledge—the specific names, dates, or codes—is concentrated in a small fraction of tokens.
- Less is More: APD doesn't need to intervene everywhere. Figure 3 shows that the gate only needs to open for a small percentage of tokens (the actual facts) to recover almost all the performance gains.
Key Results
APD consistently outperformed all baselines, including standard OPD and various teacher-forcing methods.
- Massive Gains: APD improved over standard OPD by up to 54.6%.
- State-of-the-Art: It achieved the best closed-book scores on QASPER (0.558 F1) and BioASQ (0.486 MRR).
- Efficiency: APD reaches parity with the strongest off-policy baselines while maintaining the benefits of training on student-generated states.
Limitations and Future Work
- Memory Overhead: APD requires the teacher model to stay resident in memory alongside the student during rollout generation. This could limit scaling to very large models or extremely long contexts on hardware with limited VRAM.
- Generalization: The study focused on 8B-parameter models. It remains to be seen how these bottlenecks shift as models get significantly larger or smaller.
- Linguistic Structure: The synthetic testbed deliberately removed linguistic structure to isolate variables; future work could explore how natural language patterns interact with these distillation bottlenecks.
Summary updated