Hard Negatives as Attack Directions for Cheap Robust Self-Supervised Distillation

Adversarial training makes a network robust to small worst-case input perturbations. It is expensive because every training step first runs an inner search, usually 10 or more gradient steps, for the most damaging perturbation. In self-supervised robust learning, where no labels are available, that search runs on top of an already costly contrastive objective, and training a ResNet-18 on CIFAR-10 can take over 12 GPU days. This note proposes a way to make the inner search almost free in the two-stage setting.

In the second stage, a frozen, non-robust teacher encoder already provides a fixed map of which images each image is most easily confused with, its hard negatives. The main job of an adversary is to push an image's representation toward exactly those confusable images. So we can replace the untargeted multi-step search with a single gradient step aimed at a pre-mined hard negative. The same map lets us choose how hard those targets should be.

This lets us test a tension that runs through the research file. Hard-negative work says the hardest distractors sharpen representations. Adversarial-training work says fitting the hardest instances causes robust overfitting. The question is whether the right amount of hardness, used as attack direction rather than as a loss term, can recover multi-step robustness at single-step cost.

Core Claim

Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness (DeACL) splits robust self-supervised learning into two stages. First, it runs ordinary contrastive pretraining. Second, it distills into a student whose clean and perturbed features must both match the frozen teacher's feature vector. This cuts cost from multiplicative to additive: 0.45 GPU days versus 12.6 for AdvCL, with 45.31% AutoAttack accuracy on CIFAR-10. Its second stage still uses multi-step projected gradient descent (PGD), so the inner-loop multiplier remains. That second stage is our target.

Because the teacher is frozen, every training image's nearest neighbors in teacher-embedding space can be computed once, before distillation starts, at the cost of one forward pass over the dataset. We claim that a single signed-gradient step aimed at the teacher embedding of a mined neighbor gives a stronger training adversary than an untargeted single step. We also expect it to be more stable, closing most of the gap to multi-step PGD. The attack is:

$$\delta_i = \epsilon \,\operatorname{sign}\big(\nabla_{\delta} \cos\big(g_\theta(x_i+\delta),\, t_{j}\big)\big|_{\delta=\delta_0}\big)$$

The symbols are as follows. $g_\theta$ is the student encoder. $x_i$ is the training image. $\epsilon$ is the perturbation radius. $\delta_0$ is a small random start. $t_j$ is the frozen teacher embedding of neighbor $j$, drawn from a rank window $[a,b]$ of $x_i$'s teacher-space nearest neighbors. The outer loss is DeACL's unchanged objective evaluated at $x_i+\delta_i$.

The window $[a,b]$ plays the role of the hardness parameter $\beta$ in Contrastive Learning with Hard Negative Samples, and it is the main object of study.

We make two predictions. First, with a fixed destination, the attack objective is a smoother function of the input than the untargeted "move away from yourself" objective. We therefore expect gradient alignment to stay high, the local-linearity diagnostic of Understanding and Improving Fast Adversarial Training, and catastrophic overfitting to be delayed or avoided without paying for that paper's extra-gradient regularizer. Second, robust accuracy will peak at an intermediate window. The very nearest neighbors are often false negatives (same latent class), which make weak attacks. They are also ambiguous points, and On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Training links fitting ambiguous, hard instances to large Lipschitz constants and robust overfitting. Neighbors that are too far away give easy attacks, which contribute little.

Relation to Prior Work

Three papers already combine contrastive learning with adversaries, and each plays a different role here.

Adversarial Self-Supervised Contrastive Learning (RoCL) builds attacks that push an instance toward the other instances in its batch. That is the intuition we reuse. However, RoCL does this with full PGD against all batch negatives, easy ones included, inside a coupled min-max loop, at 3.18 GPU days.

Contrastive Learning with Adversarial Examples (CLAE) uses one-step attacks to create harder training pairs. It evaluates only clean linear readout, never robustness, and has no safeguard against catastrophic overfitting.

DeACL supplies the pipeline we modify.

On the hard-negative side, Hard Negative Mixing for Contrastive Learning (MoCHi) and the importance-sampling method of Robinson et al. show that hardness can be injected at negligible cost in feature space. We borrow their hardness control, but we apply it to where the attack points, not to which negatives appear in the loss. Hard Negative Examples are Hard, but Useful is a constraint. It warns that the hardest negatives entangle features and destabilize optimization on the unit sphere, which is one more reason to expect an interior optimum in the rank window.

Fast Adversarial Training against Textual Adversarial Attacks is motivating evidence. It found that single-step and multi-step perturbation directions agree above 94% in continuous embedding spaces, which suggests the direction of the attack is what matters.

Research Program

We first reproduce DeACL on CIFAR-10 with ResNet-18: a SimCLR teacher, followed by the second stage with PGD. From the frozen teacher we precompute a 100-nearest-neighbor index over the training set.

We then implement the targeted single-step student training. It has three settings. The window $[a,b]$ is swept over ranks 1–5, 5–20, 20–100, and uniform random non-neighbors. The target is resampled from the window at every step. The radius is $\epsilon \in \{8/255, 16/255\}$.

Throughout, we log three signals for every run: gradient-alignment cosine, adversarial loss reached relative to a 10-step PGD reference on the same student, and AutoAttack accuracy per epoch. Together these separate three questions. Is the targeted step a stronger attack? Does it keep the loss surface locally linear? Does it cause robust overfitting?

As a secondary analysis, we test whether false negatives drive the failure at small ranks. We compare against a variant that excludes neighbors which a held-out labeled probe assigns to the anchor's class. This is an oracle, used only for diagnosis, in the spirit of positive-unlabeled debiasing.

Baselines

The naive reference point is DeACL's second stage with untargeted single-step attacks from a random start (FGSM-RS) at the same two passes per example. This isolates whether a target direction adds anything beyond "one step is cheaper."

The strongest existing method on this exact question is DeACL itself with multi-step PGD in the second stage. It reports 45.31% AutoAttack accuracy on CIFAR-10 in 0.45 GPU days, per Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness. We rerun it with the paper's step count, same teacher, same augmentations, same epochs, and same seed set.

Beating it means one of two things. The first is matching its AutoAttack accuracy within one point at no more than 40% of its second-stage wall-clock, counted in forward-backward passes. The second is exceeding it at equal wall-clock, where DeACL is given its best step count within that budget.

The strongest fast competitor is untargeted FGSM with the GradAlign regularizer from Understanding and Improving Fast Adversarial Training, inserted into the second stage. That paper reports it prevents catastrophic overfitting at $\epsilon = 16/255$ on CIFAR-10 with robustness comparable to PGD. No reported number was found on Lacuna for GradAlign inside self-supervised distillation. GradAlign costs extra gradient evaluations, so we compare at equal pass budget. We also report GradAlign added on top of our targeted attack.

Two ablations isolate the key ingredient. The first is a random-target attack: a single step toward the teacher embedding of a uniformly random image. This tests whether hardness, not merely having a target, matters. The second is a PGD-2 baseline at matched cost.

For grounding only, we report the 51.17% AutoAttack accuracy of supervised transferable adversarial training (ATTA) with self-adaptive targets from On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Training. It uses labels, so it is not a head-to-head comparison.

All methods get identical hyperparameter search budgets over step size and the loss-weight on the clean-versus-adversarial term.

Why not just…?

Why not just drop GradAlign into DeACL's second stage? This is the obvious combination and a real threat. If untargeted FGSM with GradAlign matches our targeted attack at equal passes, then the target direction is unnecessary and the contribution shrinks to a negative result. The equal-pass comparison settles it. Our specific bet is that the targeted objective keeps gradient alignment high on its own. That can be checked directly by logging the alignment cosine for targeted FGSM without the regularizer, at $\epsilon = 16/255$, where untargeted FGSM and FGSM-RS are known to collapse to near 0%.

Why not just run PGD with two or three steps? Truncated PGD at matched cost is a strong, simple competitor, and it is included as a baseline. The targeted method must beat PGD-2 at equal passes to justify itself. If it does not, we will report that the teacher's neighbor map offers no advantage over simply taking fewer steps.

Why not just add hard negatives to the first stage with MoCHi or importance weighting? Those methods speed up and improve the teacher. They do nothing to the adversarial inner loop, which is where the remaining multiplier sits. They are complementary rather than competing. We run a single check with a MoCHi teacher to confirm the gains add rather than overlap.

Why not just early-stop to avoid robust overfitting from hard targets? Early stopping would hide the effect of target hardness rather than explain it. We report both best-epoch and final-epoch AutoAttack accuracy, so any gain that exists only because we stopped early is visible.

Proposed Experiments

The main runs use CIFAR-10 and CIFAR-100 with ResNet-18 and SimCLR teachers, following DeACL's protocol. We also report STL-10 to test a higher-resolution setting. Evaluation uses linear probing and the robust linear evaluation of RoCL (an adversarially trained linear probe on frozen features). We report AutoAttack accuracy as standardized by RobustBench: a standardized adversarial robustness benchmark, clean accuracy, and GPU hours. We also report transfer from CIFAR-100 to CIFAR-10, following RoCL's 18.6% robust-transfer setting.

There are three decisive checks. The first is the equal-budget comparison against DeACL-PGD and GradAlign described above. The second is the hardness curve: AutoAttack accuracy as a function of rank window at both radii. The overfitting explanation predicts an interior peak together with larger best-to-final degradation at ranks 1–5. The third is the stability test at $\epsilon = 16/255$, where catastrophic overfitting would show up as a drop in gradient alignment followed by a collapse in robust accuracy.

A clear failure on either of the first two checks would falsify the core claim. That failure would still be informative about whether attack direction or attack strength drives robust distillation.