Better Answers, Weaker Evidence Control: Evidence Bypass in On-Policy Distillation

Ruoran Xu, Yilun Zhao, Ziyu Chen, Xiaoqiang kang, Arman CohanarXiv 2026 · arXiv

TL;DR

On-policy distillation (OPD) helps smaller models mimic larger ones, but it has a hidden cost: models often learn to ignore supporting evidence in favor of their own internal biases. This paper identifies this "evidence bypass" phenomenon and introduces Evidence-Paired OPD (EP-OPD), a training method that uses matched document pairs to force models to remain responsive to the text they are actually reading.

What Problem Does This Paper Address?

In the world of Large Language Models (LLMs), we often use "distillation" to transfer the reasoning capabilities of a massive "teacher" model (like Qwen-30B) to a smaller, more efficient "student" (like Qwen-4B). On-policy distillation (OPD) is a popular flavor where the student generates its own answers, and the teacher provides feedback on those specific paths.

While OPD makes students more accurate on standard benchmarks, the authors found a troubling side effect: the models become "lazy." Instead of carefully reading the provided document to find an answer, they start relying on their internal "priors"—information they learned during pre-training.

The researchers call this evidence bypass. If you take a scientific paper and subtly edit a key fact (a "counterfactual" intervention), a standard distilled model will often ignore the change and confidently repeat the answer it gave for the original, unedited document. In scientific QA, where the document is supposed to be the "ground truth," this lack of responsiveness is a major reliability failure.

Figure 2: Minimal evidence interventions identify bypass; paired evidence repairs it. One answer-critical edit flips the supported answer while the question and document structure stay fixed. Standard OPD sees only Dorig and may preserve aorig

Methods

The authors first developed a rigorous framework to measure this behavior using matched document interventions. They took questions from scientific datasets and created pairs: the original document ($D^{orig}$) and a counterfactual version ($D^{cf}$) where one answer-critical fact was changed (e.g., changing "improves" to "worsens").

To fix the bypass issue, they proposed Evidence-Paired OPD (EP-OPD). The key innovations are:

  1. Paired Training: During training, the student is exposed to both the original and the counterfactual document for the same question.
  2. Answer-Level Margin Loss: Beyond just mimicking the teacher's token-by-token distributions, the student is trained with a specific objective ($L_{pair}$) that encourages it to favor the answer supported by the current document over the answer supported by its paired alternative.

This forces the model to look at the document to decide which answer is correct, rather than relying on a "shortcut" based only on the question.

Datasets

The study focused on scientific document question answering, using two primary datasets:

  • QASPER: 5,049 questions based on 1,585 NLP papers.
  • PeerQA: 579 questions derived from peer reviews of 208 papers.

These datasets are challenging because they require precise grounding in technical text. The authors also generated several control versions of these documents, including paraphrases, irrelevant insertions, and versions where evidence was deleted entirely.

Key Insights and Ideas

  • The Accuracy-Responsiveness Trade-off: Standard distillation is a "compression" process that can inadvertently strengthen a model's internal attractors. As the model gets better at predicting the "right" answer for the training set, it gets worse at adapting when the evidence changes.
  • Priors are Attractors: The "bypass" effect is strongest when the student model already had a strong preference for the original answer before distillation. Distillation essentially "locks in" these pre-existing biases.
  • Teacher Privileged Information: One of the most effective ways to improve a student is to give the teacher extra info (like the exact span of text containing the answer) during distillation. This "Strong PI" leads to the best joint improvement in accuracy and evidence grounding.
  • Evidence Responsiveness is a Distinct Metric: The paper argues that we shouldn't just measure "Original Accuracy." We need to measure "Counterfactual Accuracy" (CFAcc) and "Bypass Rate" to truly understand if a model is reasoning or just memorizing.
Figure 1: A coordinate view of how ordinary OPD choices affect document use. We vary supervision density, teacher target, divergence, and teacher privileged information one factor at a time around standard OPD (dashed contour). Each contour plots one metric’s change (pp): original accuracy and the counterfactual response, evidence adherence, and claim grounding of the three panels of Figure 3; outward is better. Teacher and density changes move the three evidence metrics almost identically, so those contours overlap. PI (†): teacher privileged information.

Key Results

The quantitative results highlight the "repair" capability of EP-OPD:

  • Standard OPD Failures: On QASPER, standard OPD increased original accuracy by 3.4 percentage points but caused counterfactual accuracy to drop by 5.3 points and the bypass rate to spike by 15.7 points.
  • EP-OPD Success: EP-OPD recovered 54–66% of the counterfactual accuracy lost by standard OPD.
  • Preserving Gains: Crucially, EP-OPD achieved this while keeping 80–85% of the original accuracy gains provided by distillation.
  • Better Grounding: EP-OPD also improved "Claim Support Precision"—the likelihood that the individual claims made in an answer are actually supported by the cited text.
One-axis OPD interventions trace different accuracy–evidence frontiers.

Limitations and Future Work

  • Data Requirements: EP-OPD requires paired, validated counterfactual documents. While the authors built an automated pipeline for this, generating high-quality counterfactuals for every domain remains a challenge.
  • Factuality vs. Responsiveness: The authors note that their method improves responsiveness to the provided text, but it doesn't guarantee factuality if the source document itself contains errors or adversarial misinformation.
  • Internal Mechanisms: This study focuses on behavioral changes. Future work could investigate the internal "mechanistic" changes in the model's weights to see exactly how the balance between parametric knowledge and context-following shifts during distillation.

Summary updated

Abstract

On-policy distillation (OPD) can improve answer accuracy while weakening responsiveness to supporting evidence. We investigate this separation in scientific-document question answering through matched document interventions that change the supported answer while preserving the question and unrelated context. We identify evidence bypass: a student retains an originally correct answer after the document changes to support a different one. On QASPER with a Qwen3-4B student, standard OPD raises original-document accuracy by 3.4 percentage points, lowers counterfactual accuracy by 5.3 points, and increases bypass by 15.7 points; the same directional pattern appears on PeerQA. Controlled perturbations localize the shift to answer-critical evidence, and bypass grows with the student's pre-distillation closed-book preference for the original answer. A factorized study of teacher targets, privileged information, divergence, and supervision density finds that explicitly supplying the teacher with annotated answer-critical spans yields the largest joint improvement in evidence response and claim support. These findings motivate Evidence-Paired OPD (EP-OPD), which combines on-policy supervision on matched documents supporting different answers to the same question with an answer-level margin that favors each document's supported answer over its paired alternative. Across both datasets and two student sizes, EP-OPD recovers 54-66% of standard OPD's counterfactual-accuracy loss while preserving 80-85% of its original-document accuracy gain. Our results establish evidence responsiveness as a distinct training and evaluation target alongside answer accuracy. Our code is publicly available at: https://github.com/RuoranXu/EP-OPD.
BibTeX
@misc{xu2026better,
  author = {Ruoran Xu and Yilun Zhao and Ziyu Chen and Xiaoqiang kang and Arman Cohan},
  title = {{Better Answers, Weaker Evidence Control: Evidence Bypass in On-Policy Distillation}},
  year = {2026},
  eprint = {2611.01967},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01967},
}
Figure