Large self-supervised vision models such as DINOv2 hold up well on blurred, noisy, or weather-degraded images. The small models distilled from them do not. Distillation means training a small "student" network to reproduce a large "teacher" network's outputs. Current methods for distilling these teachers recover clean accuracy, but Preserving Angles Improves Feature Distillation shows they lose much of the teacher's robustness and its ability to flag unfamiliar inputs. No one has measured what happens to robustness against common corruptions, the kind catalogued in ImageNet-C, during this compression.
We propose a simple use of the teacher. Show it a clean image and a corrupted copy, and compare its two internal representations, region by region. Where the teacher's features barely move, the corruption is recoverable, and the teacher has proven that ignoring it is possible. Where the features move a lot, the information is plausibly destroyed. We use this per-region agreement, which costs nothing to compute, to decide how hard the student must match the teacher's clean features when it sees the corrupted image. The student learns the invariances the teacher actually has. It spends no capacity copying the teacher's failures or chasing clean targets that cannot be recovered.
Core Claim
Our claim is that a robust teacher's feature displacement under corruption is a better recoverability signal than an uncertainty estimate the student learns itself. Gating distillation by this displacement should let a compact student keep markedly more of the teacher's corruption robustness than three alternatives:
- clean-image feature distillation alone;
- distillation on corrupted images toward the teacher's corrupted features;
- distillation toward clean features with a learned uncertainty weight.
The gain should hold on corruption families never seen in training.
The idea also takes a position on an unresolved design choice in the literature. Some methods feed the teacher clean inputs and the student corrupted ones:
- Self-feature Distillation with Uncertainty Modeling for Degraded Image Recognition does this.
- So does HiCD.
- ARDIR argues that clean inputs to a robust teacher give the most stable targets.
Others feed corrupted inputs to both. On the benefits of knowledge distillation for adversarial robustness argues that teacher predictions on perturbed inputs carry decision-boundary information. Our gate says the right target depends on the region, and that the teacher can tell us which case applies.
How It Relates to Prior Work
Several papers motivate the gap.
- The robustness plateau. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations shows that architectural progress improved clean accuracy without improving relative corruption error. Robustness therefore has to be transferred deliberately; it does not arrive with capacity.
- Inherited brittleness. ImageNet Pre-training Also Transfers Non-robustness shows that brittle features are inherited through transfer. It leaves open whether distilling from a robust teacher breaks that inheritance.
- The teacher is a credible source. DINOv2 trains its own small variants by distilling from a 1-billion-parameter teacher and lists ImageNet-C as an evaluation. The file gives no ImageNet-C number for it, so its corruption robustness is something we must measure ourselves.
Several papers supply method.
- Angle preservation (CosPress). The angle-preserving compression of Preserving Angles Improves Feature Distillation is our backbone. It adds a trainable projection on the teacher's side, shrinking the teacher's features to the student's width. It then matches cosine-similarity structure within each batch rather than raw coordinates. This method transfers out-of-distribution detection, but it trains and evaluates only on clean images. We keep it intact and add corrupted views with the teacher-derived gate.
- Why corrupted views transfer geometry. Knowledge Transfer with Jacobian Matching proves a first-order equivalence. Matching outputs under small Gaussian input noise amounts to matching outputs plus their input gradients, so matching on corrupted views transfers the teacher's local geometry. The same paper reports vanishing second-order gradients in deep networks. We therefore use finite, sampled corruptions and never differentiate through input gradients.
- Avoiding learned discriminators. Jacobian Adversarially Regularized Networks for Robustness regularizes input gradients through a learned discriminator. We avoid that kind of adversarial min-max training for the same stability reasons.
Several papers constrain the design.
- Raw Euclidean matching is risky. Robust Vision-Language Models via Manifold-Adversarial Adapters shows that squared-error alignment of corrupted features to clean ones can push features into regions the model never sees on clean data. On their R-Bench-Dis benchmark this lowered the score from 58.79 to 57.98. This is why we match directions, not raw coordinates. It is also why we never force alignment in regions where the teacher itself cannot align.
- Closest prior method (ULFD). Self-feature Distillation with Uncertainty Modeling uses a single ResNet-50 as its own teacher, comparing its features on clean and corrupted copies. It adds a learned module that predicts per-feature variance and down-weights badly corrupted regions; its loss is called uncertainty-learning feature distillation (ULFD). Our proposal differs in two ways. The recoverability signal comes from a stronger external teacher rather than a module the student trains on itself. The setting is cross-architecture compression of a foundation model rather than self-distillation.
- Analytical marginalization does not apply here. Marginalized Denoising Auto-encoders averages over infinitely many corruptions in closed form using a second-order Taylor expansion. Its own analysis says the approximation degrades under strong noise and deep nonlinearities. Our corruptions are blur, weather, and compression, not small Gaussian noise. We therefore treat this route as ruled out for this setting.
Research Program
Setup. The teacher is a frozen DINOv2 ViT-L/14 or ViT-g/14. For each training image $x$ we draw a corrupted copy $\tilde{x}$ from a training pool restricted to two ImageNet-C families: noise and blur. Weather and digital corruptions, plus the four extra unseen corruptions, are held out entirely. This guards against the overfitting-to-benchmark failure that Benchmarking Neural Network Robustness warns about.
The gate. Let $P$ be the CosPress teacher-side projection. Let $t_i(x)$ be the projected teacher feature for patch token $i$, and $s_i(\tilde{x})$ the student's feature for the same patch on the corrupted image. Both are unit-normalized. The gate for patch $i$ is
$$w_i = \max\big(0, \cos(t_i(x), t_i(\tilde{x}))\big)^{\gamma},$$
where $\gamma$ controls how sharply the gate separates recoverable patches from destroyed ones.
The loss. The corruption term is
$$\mathcal{L}_{\text{gate}} = \frac{\sum_i w_i \,\big(1 - \cos(s_i(\tilde{x}), t_i(x))\big)}{\sum_i w_i}.$$
The total loss is the unchanged CosPress objective on clean images plus $\lambda \mathcal{L}_{\text{gate}}$. The gate is computed with gradients stopped, and it adds only one extra teacher forward pass per corrupted view.
Analysis. Beyond accuracy, we will test whether the gate tracks anything real. For each corruption type we will bin patches by $w_i$ and record how much downstream accuracy or segmentation quality the trained student recovers in each bin. If the gate is a meaningful recoverability signal, recovery should increase with $w_i$.
Students. Two students test cross-architecture transfer:
- ViT-Ti/14, matching the CosPress setting;
- ResNet-50, matching the self-feature-distillation setting so that its numbers are comparable.
The ResNet-50 student cannot use patch tokens directly. For it, the gate is computed on teacher patch tokens and resampled to the CNN's feature-map grid.
Baselines
Naive reference points. These are, first, a ResNet-50 and a ViT-Ti/14 trained normally, and second, the official DINOv2 distilled ViT-S, if a ViT-Ti is not available. They show where an undistilled or standardly distilled model sits. Benchmarking Neural Network Robustness reports an ImageNet-C mean corruption error (mCE) of 76.7 for standard ResNet-50. Mean corruption error is error averaged over corruption types and severities, normalized so that AlexNet scores 100.
CosPress on clean images only. This is the strongest existing method for distilling a foundation model while keeping robustness-related behavior. On a ViT-Ti/14 student it reports 76.6% ImageNet-1K linear-probe accuracy and 91.03% Far-OOD detection AUROC on OpenOOD (Preserving Angles Improves Feature Distillation). No reported result was found on Lacuna for CosPress on ImageNet-C or ImageNet-P, so we will measure it. Beating it means lower held-out-family mCE and lower ImageNet-P flip rate. It also means no more than half a point lost on clean linear-probe accuracy, and Far-OOD AUROC staying within a point of the CosPress value as we reproduce it.
Self-feature distillation with uncertainty modeling. This is the strongest corruption-specific competitor. It reports 46.37% mCE and 76.23% clean accuracy on ResNet-50, against QualNet's 50.34% (Self-feature Distillation with Uncertainty Modeling). Its training corruptions may overlap ImageNet-C families. We will therefore rerun it under our held-out-family protocol, and also compare against its published number under its own protocol. Beating it means a lower held-out-family mCE for the ResNet-50 student under identical corruption pools.
Ablations. Four ablations isolate the gate. All share the same CosPress backbone, corruption pool, number of corrupted views, epochs, and a fixed sweep of eight settings over $\lambda$ and $\gamma$ (or the analogous weight):
- corrupted views with uniform weights toward clean targets $t_i(x)$;
- corrupted views with uniform weights toward corrupted-teacher targets $t_i(\tilde{x})$;
- the gate replaced by ULFD's learned student-side variance;
- the gate replaced by random weights with the same mean.
Why not just…?
Why not just use CosPress as is, since angle preservation already transferred OOD detection? This is a genuine possibility. Angle preservation may carry the teacher's corruption invariance for free, because nobody has checked it on ImageNet-C. The clean-only CosPress row settles it directly. If its held-out mCE already matches the teacher-gated student, the gate is unnecessary. That would still be a useful finding in its own right.
Why not just add corruption augmentation and distill toward the teacher's features on the corrupted image? This is the most serious threat. It is ordinary distillation on augmented data, it needs no gate, and by the Jacobian-matching argument it already transfers local geometry. Our claim is that a capacity-limited student does better by concentrating on invariances the teacher demonstrably has, rather than also reproducing the teacher's drift on destroyed regions. Ablation 2 tests exactly this under an identical budget. If it ties the gated version on held-out families, we will report that the clean-versus-corrupted target question resolves toward corrupted targets for foundation teachers.
Why not just learn the uncertainty in the student, as ULFD does? A learned variance module is trained by the same student whose features are drifting. It can therefore learn to label hard-but-recoverable regions as hopeless, whereas the teacher's displacement comes from a model that has already solved those regions. Ablation 3 compares the two signals. The patch-bin recovery analysis shows which signal better predicts what the student can actually recover.
Why not just put an image-restoration network in front of a normally distilled student? Robust Vision-Language Models via Manifold-Adversarial Adapters reports about 800 ms of added latency for two-stage restoration versus about 17 ms for in-network adaptation. That latency defeats the purpose of a compact student. Simple preprocessing also helps little: contrast equalization (CLAHE) moved ResNet-50 mCE only from 76.7 to 74.5 (Benchmarking Neural Network Robustness). We will still include an off-the-shelf denoiser in front of the CosPress student as a reference row, reporting both mCE and latency.
Proposed Experiments
Benchmarks and metrics. The primary metric is ImageNet-C mCE, reported separately for the trained noise and blur families, the held-out weather and digital families, and the four unseen corruptions. Secondary metrics are:
- ImageNet-P mean flip rate, for which Benchmarking Neural Network Robustness reports a 15.6% adjacent-frame flip probability for ResNet-18 on scaling sequences;
- ImageNet-A, ImageNet-R, and ImageNet-V2 accuracy;
- OpenOOD Far-OOD AUROC and false-positive rate (FPR);
- ImageNet-1K linear-probe accuracy.
We also report a robustness-retention ratio: student held-out corruption accuracy divided by teacher held-out corruption accuracy. This separates "the student is robust" from "the teacher was robust".
Decisive checks. The proposal succeeds only if all of the following hold:
- On held-out families, the gated student beats both uniform-weight ablations and the ULFD-weight ablation, across three seeds, for both student architectures.
- The random-weight control does not reproduce the gain.
- Per-patch recovery rises with the gate value.
- Clean accuracy and OOD detection stay at CosPress levels.
If the gain appears only on the trained families, the method is merely another augmentation recipe, and we will say so.
References
- Preserving Angles Improves Feature Distillation
- Self-feature Distillation with Uncertainty Modeling for Degraded Image Recognition
- HiCD: Change Detection in Quality-Varied Images via Hierarchical Correlation Distillation
- ARDIR: Improving Robustness using Knowledge Distillation of Internal Representation
- On the benefits of knowledge distillation for adversarial robustness
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
- ImageNet Pre-training Also Transfers Non-robustness
- DINOv2: Learning Robust Visual Features without Supervision
- Knowledge Transfer with Jacobian Matching
- Jacobian Adversarially Regularized Networks for Robustness
- Robust Vision-Language Models via Manifold-Adversarial Adapters
- Marginalized Denoising Auto-encoders for Nonlinear Representations