When a large language model is compressed into a small one by distillation, which means training the small "student" to copy the outputs of the large "teacher," every sentence the teacher writes is treated as a correct target. That is the expert optimality assumption from imitation learning: the demonstrator is assumed never to be wrong. Teachers do get things wrong. Human raters found hallucinations in about 19.5% of ChatGPT responses on HaluEval (HaluEval), and larger models repeat human misconceptions more often than smaller ones do (TruthfulQA). A student distilled from such a teacher therefore spends part of its limited capacity memorizing the teacher's false claims.
This proposal borrows a fix from offline imitation learning, the setting where a policy learns from logged demonstrations without interacting with the environment. There, mixed-quality demonstrations are handled by weighting each action by how much better than expected it turned out (Exponentially Weighted Imitation Learning for Batched Historical Data). We bring that weighting to language model distillation at the level of individual reasoning steps and factual claims, and we pay for it once, offline, so inference cost does not change. The claim we test is specific. Weighting the student's copying loss step by step, using a factuality advantage estimated by rollouts, produces students that hallucinate less than students trained with uniform distillation, with sequence-level filtering, or with inference-time verification at matched compute. We further predict that the benefit grows as the student shrinks.
Core Claim
A teacher's answer is usually mostly right, with the error concentrated in a few steps: one wrong entity, a fabricated citation, a single bad calculation. Distillation methods that inherit the expert optimality assumption copy all of these steps equally. Bellman Distill (Language Model Distillation: A Temporal Difference Imitation Learning Perspective), DAgger-style ImitKD (Autoregressive Knowledge Distillation through Imitation Learning) and DistiLLM-2 (DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs) all fall in this group. Sequence-level filtering has the opposite problem: it throws away the many good steps in an answer that contains one bad one.
We weight each teacher step $s_t$ by an estimate of how much it changes the chance that the answer ends up factually correct. The value $\hat V(\cdot)$ is the fraction of $K$ teacher continuations, sampled from a given prefix, that reach a verifiably correct final answer or claim. Writing $x$ for the prompt and $s_{<t}$ for the steps before $s_t$, the step advantage is
$$\hat A_t = \hat V(x, s_{\le t}) - \hat V(x, s_{<t}),$$
and the student $\pi_\theta$ minimizes
$$\mathcal{L}(\theta) = -\mathbb{E}\Big[\textstyle\sum_t \min\big(e^{\beta \hat A_t / c},\, w_{\max}\big)\, \log \pi_\theta(s_t \mid x, s_{<t})\Big].$$
Here $\beta$ sets how aggressively bad steps are down-weighted, $c$ normalizes advantages within a batch, and $w_{\max}$ clips the weights. The clipping addresses a known failure of value-weighted imitation: noisy value estimates can produce extreme weights and destabilize training.
The objective is MARWIL's exponential advantage weighting. What changes is the advantage. In MARWIL it comes from a learned return model on game rewards. Here it is a rollout-based factuality signal on teacher text. The loss is applied at the granularity where hallucinations actually occur.
The prediction about student size comes from Hallucination is a Consequence of Space-Optimality. That paper proves that a memory-limited learner minimizes loss by confidently accepting a small fraction of non-facts, and that suppressing them costs memory that grows sharply. If so, every teacher falsehood the student is trained to copy takes up capacity, and that capacity is scarcest in the smallest students. Advantage weighting should therefore help the 0.5B student more than the 1.5B or 3B students. If the gain is flat across sizes, this capacity explanation is wrong.
Relation to Prior Work
The method comes from offline imitation learning. MARWIL supplies the exponential weighting. Action-value weighted distillation (Offline Behavior Distillation) argues for strict imitation at high-stakes decision points and loose imitation elsewhere, which is the intuition behind weighting factual steps rather than every token equally. RelaxDICE (Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching) shows that matching the exact frequencies of a flawed demonstrator traps the learner into copying its flaws. We use that result as motivation, not as a method, because its density-ratio machinery has only been shown in low-dimensional control.
The work closest to ours is Advantage-Guided Distillation (ADPA). ADPA also gives the student dense, token-level advantages, but it defines them as the log-probability gap between an aligned teacher and a reference teacher. That signal still trusts the teacher: a confidently wrong aligned teacher yields a positive advantage for its error. Our advantage is anchored to external verification of the outcome, not to teacher preferences.
DistiLLM-2 pushes down the student's own errors, but it pulls the student toward whatever the teacher generates. Bellman Distill fixes compounding error, where small early mistakes push the student into unfamiliar states. It does not address teacher error and was never evaluated on hallucination benchmarks.
Two lines of work constrain the design. Step-level verifiers such as FG-PRM and Adversarial Training for Process Reward Models detect step errors well but multiply inference cost when used for reranking. We therefore move verification into training. HaluEval shows that models detect their own hallucinations poorly, so the student's own confidence is not a usable advantage signal.
Research Program
We build the pipeline in three domains where final claims can be checked automatically.
The first is multi-hop factual question answering. Prompts come from HotpotQA training questions, the seed source of HaluEval's QA split, and correctness is answer match.
The second is mathematical reasoning. Prompts come from GSM8K and MATH training problems, and correctness is the final answer. FG-PRM's hallucination taxonomy is used to analyze which step types receive negative advantage.
The third is citation generation, where correctness means the reference exists. Existence is checked by search, following Do Language Models Know When They're Hallucinating References?, which reports 99–100% agreement between automated search validation and expert annotators.
The teacher is Qwen2.5-7B, matching DistiLLM-2's setup. The students are Qwen2.5-0.5B, 1.5B and 3B. We also distill a 4-bit-quantized 1.5B student to test whether weighting helps recover from quantization.
Steps are delimited as sentences or reasoning lines. For each step we draw $K$ teacher continuations ($K \in \{4, 8, 16\}$) to estimate $\hat V$. This is the main cost, and every baseline receives the same teacher-sampling budget. We analyze three things: how negative advantage concentrates within answers, which is what separates step-level from sequence-level treatment; sensitivity to $\beta$ and $K$, since value-weighted methods are known to be sensitive to both; and hallucination rate as a function of student size.
Baselines
The naive reference is uniform sequence-level distillation: supervised training on the teacher's greedy outputs with no weighting, using the same prompts and teacher. This measures how much teacher hallucination the student inherits by default.
The strongest existing distillation method in the file is DistiLLM-2. It reports that a Qwen2.5-1.5B student reached 44.82% on MATH against 42.94% for its 7B teacher. We rerun it with the same teacher, prompts and tuning budget, and we add its contrastive loss on top of our weighting as a combined variant.
ADPA+ is the closest dense-advantage competitor. On AlpacaEval it reports a 60.9% win rate for LLaMA-3.2-1B, against 39.3% for standard DPO. We reimplement it in our setting so the only difference is where the advantage comes from.
Bellman Distill is run as a compounding-error-aware but teacher-trusting baseline. It reports a 64.1% win rate against standard distillation, judged by GPT-4o-mini.
The most important baseline is outcome-filtered distillation. It spends the same $K$-rollout budget on sampling complete teacher answers and keeps only those whose final claim verifies, a rejection-sampling form of distillation.
Two ablations isolate the key ingredient. Hard step masking, which drops negative-advantage steps entirely, tests whether soft weighting matters. A shuffled-advantage control keeps the same weight distribution but assigns weights randomly within each sequence, which rules out regularization effects.
We found no reported result on Lacuna for any distilled student on TruthfulQA or HaluEval, so all hallucination numbers for baselines will be produced by us. Beating the baselines means two things. Our student must have a lower hallucination rate than outcome-filtered distillation and DistiLLM-2 at every student size. It must also stay within noise of DistiLLM-2 on MATH accuracy and general instruction win rate.
Why not just…?
Why not just filter whole teacher answers with a verifier and distill on the survivors? This is the real threat. If teacher errors are spread across most of an answer rather than concentrated in a few steps, filtering whole answers loses little, and our step-level weighting adds nothing. The matched-budget comparison against outcome-filtered distillation settles this directly. The localization analysis explains the result: it measures what fraction of failed answers have correct values $\hat V$ for most of their prefix. If filtering ties our method, we will report that step-level credit is unnecessary and narrow the claim to the cases where it wins.
Why not just use a bigger or more truthful teacher, or ground-truth data? TruthfulQA shows that truthfulness can fall as models grow: GPT-3-175B reached only 58%, against 94% for humans. A bigger teacher therefore does not remove imitative falsehoods. Ground-truth labels cover only the narrow answer slots of benchmark datasets. Distillation is valuable because it supplies the full reasoning text that such labels lack. Our method still uses ground truth, but only to score the teacher's steps.
Why not just verify the student's outputs at inference time with a process reward model, a model that scores each reasoning step, or with self-consistency checks? That approach multiplies the deployed cost by the number of candidates, which defeats the purpose of having a small model. HaluEval reports ChatGPT detecting hallucinations at only 62.59% accuracy in QA and 58.53% in summarization, with open models near chance, so self-checking is weak. We still test this directly. We compare against student-plus-reranker at matched inference FLOPs, and against our student combined with the same reranker, which may combine with our method rather than compete with it.
Why not just run DPO on the student with hallucinated answers as negatives? ADPA documents the alignment tax this causes in small models. In its comparison, standard DPO's MT-Bench score was 3.40, against 4.02 for ADPA+. DistiLLM-2 reports reward hacking when a DPO-style objective is used for distillation. We include DPO only as a secondary reference.
Proposed Experiments
Training uses the three domains above, plus databricks-dolly-15K prompts to preserve general ability. The dolly prompts carry no factual verifier, so their step weights are set to 1.
Evaluation covers four kinds of measure. For hallucination, we use TruthfulQA, which is held out entirely and scored with GPT-judge, reported at 90–96% agreement with humans; HaluEval QA and dialogue, measured as the hallucination rate of generated answers; and citation existence rate on held-out ACM CCS topics. For reasoning, we use GSM8K and MATH accuracy. For general quality, we use win rate against the uniform-distillation student. For cost, we record total distillation FLOPs and wall-clock time, and decoding latency, which is identical across students by construction.
Three checks decide the outcome. First, at matched teacher-sampling budget, step-weighted distillation must lower hallucination rates relative to outcome-filtered distillation on HaluEval QA and citations without losing MATH accuracy relative to DistiLLM-2. Second, the shuffled-advantage control must erase the gain. If it does not, the improvement comes from regularization, not credit assignment. Third, the gain must be largest for the 0.5B student and for the 4-bit student, as the rate-distortion argument predicts. A flat or reversed trend would refute the capacity account, even if the method still helps.
References
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Exponentially Weighted Imitation Learning for Batched Historical Data
- Language Model Distillation: A Temporal Difference Imitation Learning Perspective
- Autoregressive Knowledge Distillation through Imitation Learning
- DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
- Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testing
- Offline Behavior Distillation
- Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
- Advantage-Guided Distillation for Preference Alignment in Small Language Models
- FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoning
- Adversarial Training for Process Reward Models
- Do Language Models Know When They're Hallucinating References?