Probe-Gated Distillation: Letting a Student Refuse Its Teacher's Falsehoods

When a small language model is trained to imitate a large one (knowledge distillation), it copies the teacher's mistakes along with its skills. For factual questions this matters a great deal. Large models assert falsehoods fluently: GPT-4 fabricated 46.8% of 1,000 generated paper references in Do Language Models Know When They're Hallucinating References?. On questions built around common misconceptions, larger models are less truthful than smaller ones, as TruthfulQA showed. A student distilled from such a teacher learns to state the teacher's errors with the teacher's confidence. In the standard view this sets a ceiling: the student cannot become more factual than the model it copies.

We propose to test whether that ceiling can be broken, and with what signal. The idea is to read a cheap "is this true?" signal from the teacher's own internal activations while it writes each training answer. That signal comes from a linear probe, a small classifier trained on hidden states. We then use it to shrink the distillation loss on answer spans the probe flags as likely false. On those questions the student is not forced to overwrite what it already knew from pretraining. The headline claim is measurable: on held-out questions where the teacher is wrong but the untrained student was right, a probe-gated student keeps the right answer. As a result, its accuracy exceeds the teacher's own on TruthfulQA-style misconception questions. We also predict that the probe beats the obvious alternative, filtering by sampling consistency, specifically on falsehoods the teacher states consistently. Imitative misconceptions are exactly that kind of error.

Core Claim

We make two linked claims. The first is diagnostic. A substantial share of a distilled student's factual errors are inherited: the student gives the same wrong answer as the teacher far more often than chance. That share can be measured directly. The second is that inheritance can be cut at training time without gold labels for the training data. The teacher's internal truth signal is informative even when its sampled outputs are stable. Sampling-consistency checks are blind to stable errors.

Why should the student ever beat the teacher? Theory constrains the answer. In ridge-regression models of weak-to-strong transfer, On the Mechanisms of Weak-to-Strong Generalization shows two things. A student cannot outperform an over-regularized teacher (their Theorem 5). Any gain scales with how well the student's pretrained features already align with the truth (their Theorem 8). We take this literally. Gating cannot teach the student facts the teacher lacks. It can only stop the teacher from overwriting correct pretrained knowledge.

This gives a concrete upper bound on the achievable gain: the "rescuable set." It consists of evaluation questions the teacher gets wrong and the undistilled student gets right. TruthfulQA's inverse scaling result is precisely the reason to expect this set to be non-empty, since small models fall for fewer imitative misconceptions.

Relation to Prior Work

The closest work falls into two groups.

The first group motivates the gap without closing it. DRAG distills teacher-generated evidence into small models and raises Gemma-2-2b-it on MedMCQA from 42.91% to 72.44%. Its authors state that the student inherits the teacher's hallucinations, and they offer no mechanism against that. Winning Big with Small Models finds that self-training Llama-3-8B on its own outputs reduces hallucination about as well as distilling from GPT-4o. It attributes this to exposure bias, meaning that data drawn from the student's own distribution is easier to absorb. Neither paper separates inherited errors from native ones or tries to filter teacher falsehoods. From Teacher to Student: Tracking Memorization shows that distillation objectives filter verbatim memorization: reverse-KL distillation leaves a memorization fraction of 0.060 versus 0.330 for supervised fine-tuning on GPT-2 120M. That paper is about copying training text, not factual falsehood.

The second group supplies our method and also constrains it. The Truth Stays in the Family finds truthfulness concentrated in a few deep attention heads. These heads drift little under fine-tuning and correlate strongly within a model lineage (r = 0.77 to 0.98) but not across lineages. Its TruthProbe method steers those heads at inference time and raises Qwen2.5's HaluEval accuracy from 27.65% to 35.04%. We reuse its probing technique and codebase (miso-choi/TruthProbe), but apply the probe to the teacher during data generation rather than to the deployed model. This step matters because the lineage limit means inference-time steering cannot carry over to a student from another family. A training-time loss weight carries over to any architecture.

Student-side data selection is also related. Selective Reflection-Tuning selects teacher data using student-perplexity scores. Capacity-Dependent Effects of Data Selection for Reasoning shows that whether low-likelihood teacher data helps depends on student size. Both select data by learnability, not truth. A confidently stated misconception can be highly learnable.

Two negative results shape the design. First, From Theory to Decision Rule finds that noise-robust losses soften the damage from a weak teacher but do not move the point where teacher labels start to hurt. The reason is that foundation-model errors are structured, not random. This is why we use an error detector rather than a generic robust loss. Second, Semantic Self-Distillation distills a model's answer-level uncertainty into a single-pass predictor. It reports that the prompt representation saturates as a bottleneck. We therefore probe the teacher's answer-span activations, not only the prompt.

Research Program

The work has three parts.

Measuring inheritance. We measure how much of a distilled student's error is inherited. A teacher (Qwen2.5-72B, used as a distillation teacher in Capacity-Dependent Effects) answers a large pool of open-domain training questions drawn from the TriviaQA and HotpotQA training splits. Students are trained on these answers by standard token-level distillation. The students are Qwen2.5-1.5B and 7B, which share the teacher's lineage, and Gemma-2-2b, which does not.

On held-out evaluation questions we compute three quantities: the inherited-error rate (the probability that the student gives the teacher's wrong answer, given that the teacher is wrong), the student's native error rate, and the rescuable set defined above. This alone answers an open question from the file. It tells us whether TruthfulQA's inverse-scaling falsehoods survive distillation into a smaller student, or whether limited student capacity blocks them.

Building the gate. We train linear probes on teacher head activations at answer-span tokens, following the TruthProbe recipe. The probe training set is a small gold-labeled pool of a few thousand true and false teacher answers. It is kept disjoint from both the distillation pool and the evaluation sets.

For a training answer $y$ to question $x$, let $s(x,y) \in [0,1]$ be the probe's truth score. The student $p_S$ is trained with

$$ \mathcal{L} = \sum_{t \in y} w(x,y)\,\mathrm{KL}\!\left(p_T(\cdot \mid x, y_{<t}) \,\|\, p_S(\cdot \mid x, y_{<t})\right), \qquad w(x,y) = \sigma\!\left(\frac{s(x,y)-\tau}{\beta}\right), $$

where $p_T$ is the teacher's next-token distribution, $\sigma$ is the logistic function, $\tau$ is a threshold, and $\beta$ is a temperature. Both $\tau$ and $\beta$ are tuned on a validation split. Flagged answers are down-weighted, not replaced. On those questions the student keeps its pretrained behavior, which matches the Theorem 8 mechanism. A separate variant converts flagged items into "I don't know" targets, so that refusing can be distinguished from preserving pretrained knowledge.

Characterizing teacher errors. For every teacher error we also sample the teacher ten times and ask indirect follow-up questions, as in Do Language Models Know When They're Hallucinating References?. This labels each error as unstable (the samples disagree) or stable (consistent but wrong). The split lets us test exactly where a probe signal should beat a consistency signal.

Baselines

The naive reference is plain distillation. The student minimizes the unweighted objective above on all teacher answers, using the same data, the same number of steps and the same learning-rate sweep as every other arm. A second reference is the undistilled student itself, which defines the rescuable set. A third is the teacher's own accuracy, which defines the ceiling. Beating the ceiling means the gated student's held-out accuracy is above the teacher's, with a confidence interval from a paired bootstrap over questions.

The strongest competitor is consistency-filtered distillation. It drops any training answer whose ten teacher samples or indirect follow-up answers disagree, following the detection approach of Do Language Models Know When They're Hallucinating References?. It needs no labels and is the method a practitioner would reach for first. It gets ten times our generation budget, and its threshold is tuned on the same validation split. Probe gating beats it if gating gives higher held-out accuracy at matched retained-data fraction, and if the advantage is concentrated on stable teacher errors.

Second, self-training, as in Winning Big with Small Models. The student is trained on its own answers to the same questions, with the same step budget. That paper reports no scalar FactScore on Lacuna, so the comparison here is run fresh.

Third, a noise-robust loss: Generalized Cross Entropy applied to the teacher's sequence-level targets, as benchmarked in From Theory to Decision Rule.

Fourth, learnability filtering with the reversed instruction-following-difficulty score from Selective Reflection-Tuning. It retains the same data fraction as the probe gate, which isolates whether truth or learnability drives the gain.

Fifth, inference-time steering. TruthProbe is applied directly to the plainly distilled student, with probes trained on that student. On HaluEval it is the strongest reported number in the file for this kind of intervention: Qwen2.5 accuracy goes from 27.65% to 35.04% in The Truth Stays in the Family.

Two ablations isolate the key ingredient. In the first, the probe is replaced by the teacher's own answer log-probability, which tests whether the probe adds anything beyond confidence. In the second, the probe is replaced by gold correctness labels on the training data, which gives an oracle upper bound on how much any filter could help.

For external reference points: the best TruthfulQA truthfulness in the original paper is 58%, from GPT-3-175B with a helpful prompt, against roughly 94% for humans (TruthfulQA). ChatGPT's HaluEval QA detection accuracy is 62.59% (HaluEval). We found no reported result on Lacuna for any distillation method's student factual accuracy on TruthfulQA or HaluEval QA.

Why not just…?

Why not just filter teacher answers by self-consistency? This is the real threat. On unstable teacher errors we expect it to match or beat the probe. The argument for the probe rests on stable errors. Imitative misconceptions are falsehoods a large model learned from repeated human text and reproduces reliably, so sampling-based checks pass them. The error-stability split settles this directly. If consistency filtering matches probe gating on stable errors too, the probe offers only a cost advantage: one forward pass versus ten samples. We will report that outcome as such.

Why not just apply TruthProbe to the student at inference time? It works within a lineage. But The Truth Stays in the Family reports that truth-score correlations collapse across unrelated families. Steering also leaves the student's weights carrying the inherited falsehoods. The Gemma student tests the cross-family case. Comparing steering, gating, and steering on top of a gated student shows whether the two interventions are redundant or additive.

Why not just use a noise-robust loss? From Theory to Decision Rule found that these losses do not move the crossover point under structured foundation-model errors. We include Generalized Cross Entropy anyway, so that the claim is tested in the language setting rather than assumed.

Why not just train the student to say "I don't know"? Winning Big with Small Models found that refusal training helps on unanswerable queries. It does not address plausible fabrications within answerable scope. Refusal also cannot raise accuracy above the teacher's, only lower the error rate. The refusal variant of our gate separates "abstains more" from "keeps correct knowledge," and we report accuracy and abstention rate separately.

Proposed Experiments and Decisive Checks

Evaluation uses TruthfulQA generation scored by its GPT-judge, plus its multiple-choice format; held-out TriviaQA; the HaluEval QA split (HaluEval); and the reference-fabrication protocol on ACM CCS titles from Do Language Models Know When They're Hallucinating References?. We also track MMLU accuracy to confirm that gating does not cost general capability.

The first decisive check is the rescuable set. If it is near-empty for a given benchmark and student, no filter can beat the teacher there. We report this before any method comparison, so that a null result is interpretable.

The second check is the ceiling test. On TruthfulQA, where inverse scaling predicts a large rescuable set, the gated student should exceed teacher accuracy. It should also recover a meaningful fraction of the oracle-filter gain. On TriviaQA, where the teacher is broadly stronger, we predict the gate reduces inherited error without beating the teacher.

The third check is the stability interaction. Probe gating should beat consistency filtering on stable teacher errors and tie on unstable ones. A uniform tie would refute the central mechanistic claim.

The fourth check concerns probe quality across student capacity. Following Capacity-Dependent Effects, we expect gating to help the 1.5B student mainly by preserving pretrained answers. For the 7B student we expect gains on harder held-out questions as well.

The main risk is that a probe trained on TriviaQA-style facts does not recognize misconceptions. We test this before running any distillation, by measuring the probe's detection AUROC on teacher answers to a misconception-heavy held-out pool. If it falls to chance, the proposal reduces to the inheritance measurement plus the consistency baseline. That result is still informative about the teacher-dependent ceiling.