Knowledge distillation trains a small "student" network to imitate the output probabilities of a larger "teacher." Increasingly this is done offline: the teacher is run once over the training set, its outputs are written to disk, and the student is trained later from that stored table (the "cache") without the teacher in memory. Offline caching is usually treated as a speed trick. It saves about 75% of experiment time in Speedy Gonzales and gives a 2.2x speedup in FDBPL. It also has a security property that has not been examined. In the offline setting, the cache is the entire channel through which the teacher can influence the student. Every bit of knowledge a backdoored teacher passes on, including a hidden trigger response, has to be written into a finite table that the defender holds and can rewrite before the student sees a single gradient.
This matters because the most worrying recent attack travels through exactly this channel. Taught Well Learned Ill (SCAR) builds a "distillation-conditional" backdoor, meaning a teacher that shows only about 1.5% attack success itself, passes audits, and yet produces students with over 99% attack success on CIFAR-10. It does this by shaping the teacher's soft outputs on clean data so that students pick up the trigger. We propose to build the cache from several architecturally different public teachers and to store, for each training example, the coordinate-wise median of their probability vectors rather than their mean. When honest teachers are a majority, this places every cached target inside the range spanned by the honest teachers, regardless of what the malicious teacher outputs. The question is whether that bounded freedom is too small to carry a backdoor, while the cache still keeps most of distillation's accuracy benefit.
Core Claim
Let $p_i^{(t)}$ be teacher $t$'s probability for class $i$ on example $x$, for $T$ teachers of which at most $m < T/2$ are adversarial. The median cache stores
$$ q_i(x) = \mathrm{median}_{t}\, p_i^{(t)}(x), \quad \text{renormalized over } i, $$
and the student minimizes KL divergence to $q$. Before renormalization, each $q_i(x)$ lies between the smallest and largest honest value $p_i^{(t)}(x)$. That is a standard property of the median under an honest majority. The attacker's control over the student's targets therefore shrinks from the whole probability simplex, which is what it controls under single-teacher distillation, to an "honest envelope." The width of that envelope is simply how much the honest teachers disagree with each other.
We claim three things. First, on the clean and lightly poisoned data where SCAR and related attacks must operate, the envelope is narrow enough that student attack success falls to roughly the level of a student trained without any teacher. Second, mean-averaging, the default way to combine ensemble outputs, does not have this property, because a single teacher can move the mean by up to $1/T$ in any direction. Third, the defense costs one extra pass over a table the practitioner already stores. That removes the "multiple teachers are too expensive" objection that BDEKD raises against scaling ensemble defenses.
The claim can fail. An adaptive attacker who knows the aggregation rule may still find a backdoor that fits inside the envelope. Measuring how attack success scales with envelope width is the central scientific result, whichever way it comes out.
Relation to Prior Work
Two attack papers motivate this work. SCAR (Chen et al.) hides the backdoor in a single teacher's soft outputs and optimizes it against a single surrogate student using full-probability, online distillation. It never considers ensembles or cached, aggregated targets, and our adaptive attack extends its optimization to cover them. How to Backdoor the Knowledge Distillation leaves the teacher clean and instead poisons the distillation images with small adversarial perturbations that make that one teacher agree with the target label, reaching 99.2% attack success on CIFAR-10. Its perturbations are crafted against one architecture. Under a heterogeneous median they must fool a majority of different architectures at once. We test that setting as a second threat, and BackdoorBench's finding that vulnerability differs sharply across backbones gives some reason to expect it to be hard.
Three defense papers constrain the design. BDEKD distills a student from three purified copies of one compromised BERT and cuts average attack success to about 17%. Its teachers share an architecture and an origin, it runs online, and it averages alignment losses. We use independent, heterogeneous teachers, work offline, and aggregate robustly. Robust Knowledge Distillation in Federated Learning already uses a median-flavored selection of trusted clients before ensemble distillation, keeping attack success below 17% with up to 60% malicious clients. It applies robustness to model updates rather than to per-example cached targets, and it relies on clustering. This is prior evidence that robust statistics help here. NBA and Neural Attention Distillation purify a student using a clean teacher and fresh adversarial samples at every batch. They assume a trusted teacher exists and do not fit a static cache. SCAR also reports that it evades NAD.
Method components come from existing work. Soft-label caching uses the FKD codebase (szq0214/FKD, accompanying FerKD). Top-$K$ truncation follows Knowledge Inheritance, and we study it as an interacting factor because truncation discards the low-probability tail where SCAR plausibly encodes its signal.
Research Program
The threat model is a practitioner who builds a cache from $T$ public checkpoints of different architectures, for example ResNet-50, VGG-19, ViT, and EfficientNet. The practitioner does not know which checkpoints, if any, are compromised, but assumes an honest majority. The distillation images may additionally be poisoned in the way Wu et al. describe.
The work has three parts. The first is the defense itself: a median cache, optionally truncated to top-$K$, plus an entry-level disagreement score, the envelope width, which lets the defender drop the most contested examples. This is possible only offline, because it requires statistics over the whole dataset before training begins.
The second part is an adaptive attack. We extend SCAR's two-level optimization so the inner surrogate student distills from a median (or smoothed median) of the malicious teacher and surrogate honest teachers of the public architectures, trained by the attacker. This is the strongest attacker we can build, and a defense that only beats non-adaptive SCAR will not be claimed as a result.
The third part is an analysis relating student attack success to measured envelope width. We vary envelope width by changing $T$, the architectural diversity of the teachers, and the teachers' training seeds. If attack success tracks envelope width, the practical advice is concrete: add diverse honest teachers until the envelope is narrow.
Baselines
The naive reference point sits at two extremes. At one end is a student trained from scratch on ground-truth labels. It inherits no teacher backdoor and sets the attack-success floor, but gives up distillation's accuracy gain. At the other end is full-probability distillation from the single compromised teacher, which SCAR reports exceeds 99% student attack success on CIFAR-10. We found no reported scratch-student accuracy on Lacuna for these teacher and student pairs, so we will measure it. We summarize results as the fraction of the accuracy gap recovered, $(A - A_{\text{scratch}})/(A_{\text{clean-KD}} - A_{\text{scratch}})$, where $A_{\text{clean-KD}}$ is a student distilled from an all-honest ensemble. Beating the naive baseline means recovering most of this gap while keeping attack success within a few points of the scratch floor.
The strongest ensemble competitor is mean-aggregated offline distillation from the same heterogeneous teachers. It is the default practice and would plausibly dilute a single bad teacher. It is also the closest analogue of BDEKD's averaged ensemble alignment. BDEKD reports about 17% average attack success in NLP, and we found no reported result on Lacuna for heterogeneous mean-ensemble distillation against SCAR. The mean cache uses the same teachers, the same cached examples, the same student and training schedule, and the same temperature search as the median cache. The median must show lower attack success than the mean under adaptive SCAR at matched accuracy.
The strongest single-teacher defense is NAD applied after distillation. NBA reports NAD at 4.68% average attack success against conventional backdoors, and SCAR reports that NAD fails to remove its conditional backdoor. We also run NBA where compute allows, since it reports 1.52% average attack success with 5% clean data. All post-hoc defenses get the same 5% clean split.
Two ablations isolate the key ingredient. The first is a hard-label cache that stores only the median teacher's top-1 class, which removes the tail channel entirely. The second is a median cache built from homogeneous teachers of one architecture with different seeds. Comparing it with the heterogeneous version separates the effect of robust aggregation from the effect of architectural diversity.
Why not just…?
Why not just distill from hard labels or train from scratch? Removing soft targets almost certainly blocks SCAR's clean-data channel, and this is the most serious competitor. It also discards the dark knowledge that makes distillation worthwhile. The hard-label and scratch baselines are run under identical budgets. If the median cache does not recover clearly more of the accuracy gap at equal attack success, the proposal's value collapses, and we will report that outcome.
Why not just average the teachers? Averaging with one bad teacher out of $T$ still gives that teacher a $1/T$ share of every target in any direction it chooses, and an adaptive attacker can amplify its signal to compensate. The mean-versus-median comparison under the adaptive SCAR variant settles this. If averaging turns out equally robust in practice, we will report that the envelope bound is not the operative mechanism.
Why not just purify the student afterward with NAD or pruning? SCAR reports robustness to Neural Cleanse, SCALE-UP, and NAD. It also argues that pruning-style inspection fails because the teacher's dormant neurons never fire until distillation happens, which undercuts the Fine-Pruning premise in this setting. We still run these defenses, and also combine them with the median cache, to check whether they add anything.
Why not just filter the distillation data? The perturbed images of Wu et al. are imperceptible, and anomaly filtering degrades under diverse data, as the RKD paper notes. Our disagreement score is a form of filtering, but it works on teacher outputs rather than raw inputs. We compare it against dropping an equal number of examples at random.
Proposed Experiments
The main benchmarks are CIFAR-10 and the 50-class ImageNet subset used by SCAR, with GTSRB as a third. Teachers are drawn from ResNet-50, VGG-19, ViT, and EfficientNet-B3, and students are MobileNet-V2 and ShuffleNet-V2, matching SCAR's pairs. Baseline attacks come from the BackdoorBench codebase. We measure student clean accuracy, attack success, fraction of accuracy gap recovered, envelope width, and cache-building cost.
The first decisive check places one SCAR teacher among three heterogeneous honest teachers and compares mean and median caches with the full probability tail. The second repeats the comparison against our adaptive, median-aware SCAR. The third sweeps $T$ from 3 to 7 and plots student attack success against measured envelope width. The fourth crosses top-$K$ truncation with each aggregation rule to find out whether truncation removes SCAR's channel or instead concentrates the student on dominant shortcuts. The fifth runs the poisoned-data attack of Wu et al. with perturbations crafted against one teacher and, separately, against the whole ensemble.
The proposal succeeds if, under the adaptive attack on CIFAR-10, the median cache holds student attack success near the scratch floor while recovering substantially more of the accuracy gap than hard labels. It also succeeds as a result if attack success rises cleanly with envelope width, since that yields a measurable rule for how much teacher diversity a practitioner needs.
References
- Speedy Gonzales: A Collection of Fast Task-Specific Models for Spanish
- FDBPL: Faster distillation-based prompt learning for region-aware vision-language models adaptation
- Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack
- BDEKD: mitigating backdoor attacks in NLP models via ensemble knowledge distillation
- How to Backdoor the Knowledge Distillation
- BackdoorBench: A Comprehensive Benchmark of Backdoor Learning
- Robust Knowledge Distillation in Federated Learning: Counteracting Backdoor Attacks
- NBA: defensive distillation for backdoor removal via neural behavior alignment
- Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks
- FerKD: Surgical Label Adaptation for Efficient Distillation
- Knowledge Inheritance for Pre-trained Language Models
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks