TL;DR
On-policy distillation (OPD) helps smaller models mimic larger ones, but it has a hidden cost: models often learn to ignore supporting evidence in favor of their own internal biases. This paper identifies this "evidence bypass" phenomenon and introduces Evidence-Paired OPD (EP-OPD), a training method that uses matched document pairs to force models to remain responsive to the text they are actually reading.
What Problem Does This Paper Address?
In the world of Large Language Models (LLMs), we often use "distillation" to transfer the reasoning capabilities of a massive "teacher" model (like Qwen-30B) to a smaller, more efficient "student" (like Qwen-4B). On-policy distillation (OPD) is a popular flavor where the student generates its own answers, and the teacher provides feedback on those specific paths.
While OPD makes students more accurate on standard benchmarks, the authors found a troubling side effect: the models become "lazy." Instead of carefully reading the provided document to find an answer, they start relying on their internal "priors"—information they learned during pre-training.
The researchers call this evidence bypass. If you take a scientific paper and subtly edit a key fact (a "counterfactual" intervention), a standard distilled model will often ignore the change and confidently repeat the answer it gave for the original, unedited document. In scientific QA, where the document is supposed to be the "ground truth," this lack of responsiveness is a major reliability failure.
Methods
The authors first developed a rigorous framework to measure this behavior using matched document interventions. They took questions from scientific datasets and created pairs: the original document ($D^{orig}$) and a counterfactual version ($D^{cf}$) where one answer-critical fact was changed (e.g., changing "improves" to "worsens").
To fix the bypass issue, they proposed Evidence-Paired OPD (EP-OPD). The key innovations are:
- Paired Training: During training, the student is exposed to both the original and the counterfactual document for the same question.
- Answer-Level Margin Loss: Beyond just mimicking the teacher's token-by-token distributions, the student is trained with a specific objective ($L_{pair}$) that encourages it to favor the answer supported by the current document over the answer supported by its paired alternative.
This forces the model to look at the document to decide which answer is correct, rather than relying on a "shortcut" based only on the question.
Datasets
The study focused on scientific document question answering, using two primary datasets:
- QASPER: 5,049 questions based on 1,585 NLP papers.
- PeerQA: 579 questions derived from peer reviews of 208 papers.
These datasets are challenging because they require precise grounding in technical text. The authors also generated several control versions of these documents, including paraphrases, irrelevant insertions, and versions where evidence was deleted entirely.
Key Insights and Ideas
- The Accuracy-Responsiveness Trade-off: Standard distillation is a "compression" process that can inadvertently strengthen a model's internal attractors. As the model gets better at predicting the "right" answer for the training set, it gets worse at adapting when the evidence changes.
- Priors are Attractors: The "bypass" effect is strongest when the student model already had a strong preference for the original answer before distillation. Distillation essentially "locks in" these pre-existing biases.
- Teacher Privileged Information: One of the most effective ways to improve a student is to give the teacher extra info (like the exact span of text containing the answer) during distillation. This "Strong PI" leads to the best joint improvement in accuracy and evidence grounding.
- Evidence Responsiveness is a Distinct Metric: The paper argues that we shouldn't just measure "Original Accuracy." We need to measure "Counterfactual Accuracy" (CFAcc) and "Bypass Rate" to truly understand if a model is reasoning or just memorizing.
Key Results
The quantitative results highlight the "repair" capability of EP-OPD:
- Standard OPD Failures: On QASPER, standard OPD increased original accuracy by 3.4 percentage points but caused counterfactual accuracy to drop by 5.3 points and the bypass rate to spike by 15.7 points.
- EP-OPD Success: EP-OPD recovered 54–66% of the counterfactual accuracy lost by standard OPD.
- Preserving Gains: Crucially, EP-OPD achieved this while keeping 80–85% of the original accuracy gains provided by distillation.
- Better Grounding: EP-OPD also improved "Claim Support Precision"—the likelihood that the individual claims made in an answer are actually supported by the cited text.
Limitations and Future Work
- Data Requirements: EP-OPD requires paired, validated counterfactual documents. While the authors built an automated pipeline for this, generating high-quality counterfactuals for every domain remains a challenge.
- Factuality vs. Responsiveness: The authors note that their method improves responsiveness to the provided text, but it doesn't guarantee factuality if the source document itself contains errors or adversarial misinformation.
- Internal Mechanisms: This study focuses on behavioral changes. Future work could investigate the internal "mechanistic" changes in the model's weights to see exactly how the balance between parametric knowledge and context-following shifts during distillation.
Summary updated