Models trained to resist adversarial examples (tiny, deliberately chosen input changes that flip a prediction) are expensive to produce. Several of them, notably robust ImageNet backbones, are already trained and public. The cheap way to use one is to fine-tune it on a new task with ordinary clean data. Unfortunately, the robustness then disappears as soon as more than the last few layers are updated. This proposal asks whether that robustness can be kept through deep fine-tuning without ever generating an adversarial example downstream.
The proposed method adds one penalty to clean fine-tuning. It keeps the fine-tuned model's input gradients, meaning how its internal features change as each pixel changes, close to the input gradients of the robust model it started from. The motivation is that robustness is a property of local sensitivity to the input, not of the feature values themselves. Existing "don't forget" regularizers anchor the feature values, which leaves sensitivity free to drift. We expect this to cost roughly two to three times a clean fine-tuning step. That is well below multi-step attack-based training, though it is not free.
Core Claim
Fine-tuning a robust backbone on clean target data destroys inherited robustness, and the literature documents this repeatedly. Adversarially robust transfer learning transfers a robust ImageNet model with clean data only. It works when retraining the head or the last blocks, but robustness collapses when deeper layers are unfrozen, even with its Learning-without-Forgetting (LwF) feature-distillation penalty. CARTL reports the LwF baseline dropping to near 0% robustness. On the Robustness Tradeoff in Fine-Tuning shows the same pattern step by step, including for low-parameter adapters such as LoRA and BitFit: robustness peaks early, then decays as the model picks up task-predictive but non-robust features.
Our claim is that a missing constraint explains much of this. LwF matches what the features are at clean inputs. It does not constrain how they move when the input is nudged, and that movement is what an attacker exploits.
Knowledge Transfer with Jacobian Matching supplies the method. It shows that distilling under input noise is, to first order, matching outputs plus matching input-output Jacobians (the matrix of derivatives of outputs with respect to inputs). It also argues LwF is related to a distillation bound. That paper evaluated noise robustness and low-data accuracy, never white-box attacks. Robust Learning with Jacobian Regularization supplies the cost trick. A single random projection reduces Jacobian penalties to 1.3–2× a normal backward pass. However, it pushes the Jacobian toward zero, works from scratch, and never studies transfer.
We combine the two in a specific direction. The target is not zero sensitivity but the robust teacher's sensitivity. This target is the "perceptually aligned gradient" structure that Do Adversarially Robust ImageNet Models Transfer Better? identifies as the hallmark of robust ImageNet models.
Concretely, let $f_\theta(x)\in\mathbb{R}^d$ be the student's penultimate features on input $x$, let $g(x)$ be the frozen robust teacher's features at the same layer, and let $v$ be a fresh random unit vector in $\mathbb{R}^d$ drawn per example. The training loss is:
$$\mathcal{L}(\theta) = \mathrm{CE}\big(h(f_\theta(x)), y\big) + \lambda \,\big\| \nabla_x \big(v^\top f_\theta(x)\big) - \nabla_x \big(v^\top g(x)\big) \big\|_2^2$$
Here $h$ is the downstream classification head, CE is cross-entropy, $y$ is the label, and $\lambda$ is the penalty weight. Each gradient term is one vector-Jacobian product, so the student pays one extra backward pass plus the cost of differentiating through it. The teacher pays one forward and one backward pass. No inner maximization loop is run.
Scope and Honest Constraints
The method needs a robust source checkpoint. It therefore does not solve the harder problem of making a naturally trained foundation model such as CLIP or DINOv2 robust without attacks, which is Gap 3 in the file. What it targets is the realistic setting where public robust backbones exist but every downstream user currently has two bad options. They can freeze the backbone and accept poor accuracy, or they can fine-tune deeply and lose robustness. CARTL avoids that choice only by modifying source pre-training.
Downstream robustness cannot exceed what the teacher carries. This "ceiling" was noted by Adversarially robust transfer learning. Success therefore means closing the gap to the ceiling at high downstream accuracy, not beating attack-trained models in absolute robustness.
Because the method explicitly regularizes gradients, the main scientific risk is gradient masking. This is a model that looks robust to gradient-based attacks only because its gradients have become uninformative. The evaluation is built around catching that.
Research Program
We first reproduce the collapse. Robust ResNet-50 and WideResNet ImageNet checkpoints, in the setting of Do Adversarially Robust ImageNet Models Transfer Better?, are fine-tuned on clean target data in three ways: full fine-tuning, LoRA, and BitFit. We record clean and PGD accuracy at every checkpoint, mirroring the protocol of On the Robustness Tradeoff in Fine-Tuning. This yields robustness-versus-accuracy curves rather than single points.
We then add the gradient-matching penalty and study three questions. The first is where to match: penultimate features versus several intermediate layers. The second is how many random projections per example are needed. The third is whether the penalty is needed throughout training or mainly late, when the robustness-tradeoff paper says decay sets in.
As a mechanistic check, we measure the cosine similarity between the student's and teacher's input gradients during training. We test whether its decline predicts robustness loss across all methods, including LwF and clean fine-tuning. If gradient divergence tracks robustness collapse and feature divergence does not, that supports the core claim independently of the method's headline numbers.
A secondary arm moves to ViT backbones and to segmentation. We use the layer6ai-labs/ssl-robustness codebase, which implements task-specific attacks such as SegPGD. Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks found that encoder-level robustness gives almost no protection against such attacks. Our hypothesis is that anchoring the full input-gradient structure, rather than embedding invariance, carries over better to dense tasks. That finding makes this arm a genuine test rather than a formality.
Baselines
The do-nothing reference is plain clean fine-tuning of the robust backbone, both full and LoRA, with the same schedule and augmentation as our method. It sets the floor that robustness collapses to.
The frozen-backbone reference is a linear probe on robust features, trained on clean data. On CIFAR-10, Adversarially robust transfer learning reports about 88% clean accuracy and about 22% PGD-20 robustness for this style of transfer. That is the point any deep-tuning method must dominate.
The closest existing attack-free method is LwF feature distillation toward the robust teacher, following the same paper. It costs about one extra teacher forward pass. It is the most direct competitor because it differs from our method only in matching feature values instead of input gradients.
The strongest reported attack-free number in the file is from CARTL. On CIFAR-100→CIFAR-10 transfer it reports 29.34% robustness against PGD-100 at about 91% clean accuracy, versus near 0% for LwF. CARTL modifies source pre-training. We will therefore run it on its own CIFAR-100→CIFAR-10, SVHN, and GTSRB WideResNet setting as a reference, and run our method from a standardly adversarially trained CIFAR-100 WideResNet in the same setting. Beating CARTL means at least 29.34% PGD-100 robustness at matched ~91% accuracy without a modified source. Matching it would already be a useful result, since it removes the source-side requirement.
The attack-based upper reference is Adversarial Fine-Tuning from A Simple Fine-tuning Is All You Need. It runs 10 epochs of PGD with a slow-start, fast-decay learning rate and reports 88.15% clean and 51.7% PGD accuracy on CIFAR-10, at 8–10× less cost than training from scratch. It indicates how much robustness is left on the table by not attacking at all.
The cost-matched attack baseline is single-step adversarial fine-tuning, which uses one gradient-sign perturbation per batch. The file contains no reported result on Lacuna for single-step adversarial fine-tuning of robust backbones. We will run it ourselves, using the fast adversarial training implementation in alibaba/easyrobust, at a wall-clock budget equal to our method's.
Three ablations isolate the key ingredient. The first matches the student's input gradients to zero instead of to the teacher, which is Hoffman-style regularization from Robust Learning with Jacobian Regularization applied at transfer. The second is Gaussian-noise output distillation, its first-order equivalent according to Knowledge Transfer with Jacobian Matching. The third combines our penalty with LwF.
All methods share data, augmentation, and backbone, and each receives the same number of hyperparameter trials, with $\lambda$-type weights swept on a held-out split. We report wall-clock time per epoch alongside accuracy. Every comparison is made on the full robustness-accuracy curve, not a single cherry-picked checkpoint.
Why not just…?
Why not just linear-probe the robust backbone? It keeps robustness for free. However, CARTL reports "mediocre accuracy" from tuning only the last layer, and the robustness-tradeoff study finds linear probing below adapter methods on the joint robustness-accuracy curve. Our curves will show directly whether gradient matching reaches full-fine-tuning accuracy while staying above the probe's robustness. If it does not, the method has no reason to exist.
Why not just use LwF or feature distillation? It is cheaper. Its documented failure under deep fine-tuning is the very phenomenon we target. The LwF-versus-gradient-matching comparison at equal depth, together with the gradient-similarity diagnostic, settles whether matching sensitivity rather than values is what matters.
Why not just distill on Gaussian-noised inputs? This is a real threat. Srinivas and Fleuret show it is first-order equivalent to Jacobian matching, and it needs only forward passes. Their own results suggest sampled noise needs many more samples to cover the input neighbourhood than the analytic penalty. Adversarial directions are also rare in high dimensions, so random noise may miss them. We run it at matched wall-clock time. If it matches our method, we will report that the cheaper version suffices and treat that as the finding.
Why not just stop fine-tuning early, since robustness peaks early? The peak in On the Robustness Tradeoff in Fine-Tuning comes before accuracy has converged, so early stopping trades one away for the other. The full curves show whether our method moves the entire frontier or merely picks a better point on it.
Why not just run single-step adversarial fine-tuning? It has similar cost: one extra backward pass. It also targets worst-case directions, which our random projections do not. This is the most serious competitor. A cost-matched comparison under strong attacks decides it. Our method wins only if it achieves higher robustness at equal accuracy and time, or avoids the instability that single-step methods are prone to.
Proposed Experiments
The main transfer suite takes robust ImageNet backbones to CIFAR-10, CIFAR-100, Caltech-256, CUB-200, Stanford Dogs, SVHN, and GTSRB. It covers full fine-tuning and LoRA and reports clean accuracy, PGD-100 accuracy with restarts, and time per epoch. The CARTL-matched WideResNet setting provides the head-to-head number.
The decisive checks guard against gradient masking, because the method regularizes the very gradients that attacks use. We run four of them. First, attack strength is increased to many steps with random restarts and should saturate. Second, accuracy should fall to zero at large perturbation radii. Third, transfer attacks crafted on the robust teacher and on the clean-fine-tuned model should not succeed more often than white-box attacks. Fourth, a gradient-free black-box attack must not reveal robustness that gradient attacks miss.
The method is judged successful if, on at least most target datasets, it sits strictly above both LwF and cost-matched single-step adversarial fine-tuning on the robustness-accuracy curve while passing all masking checks. It fails if Gaussian-noise distillation matches it at lower cost, or if the zero-target ablation does as well. In the second case, the teacher's gradient structure is not the active ingredient.
The secondary ViT and segmentation arm reports mIoU under SegPGD. It is judged against the near-zero protection that Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks found for encoder-level defenses.
References
- Adversarially robust transfer learning
- CARTL: Cooperative Adversarially-Robust Transfer Learning
- On the Robustness Tradeoff in Fine-Tuning
- Knowledge Transfer with Jacobian Matching
- Robust Learning with Jacobian Regularization
- Do Adversarially Robust ImageNet Models Transfer Better?
- Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks
- A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning