Abstract
In this paper, we develop improved techniques for defending against adversarial examples at scale. First, we implement the state of the art version of adversarial training at unprecedented scale on ImageNet and investigate whether it remains effective in this setting. Next, we introduce enhanced defenses using a technique we call logit pairing, a method that encourages logits for pairs of examples to be similar. When applied to clean examples and their adversarial counterparts, logit pairing improves accuracy on adversarial examples over vanilla adversarial training; we also find that logit pairing on clean examples only is competitive with adversarial training in terms of accuracy on two datasets. Finally, we show that adversarial logit pairing achieves the state of the art defense on ImageNet against PGD white box attacks, with an accuracy improvement from 1.5% to 27.9%. Adversarial logit pairing also successfully damages the current state of the art defense against black box attacks on ImageNet (Tramer et al., 2018), dropping its accuracy from 66.6% to 47.1%. With this new accuracy drop, adversarial logit pairing ties with Tramer et al. (2018) for the state of the art on black box attacks on ImageNet.
Introduction
Deep learning models are notoriously brittle. Small, often imperceptible perturbations—called adversarial examples—can trick a state-of-the-art classifier into misidentifying a stop sign as a toaster. While researchers had found ways to defend models on small datasets like MNIST, these defenses historically failed to scale. Before this work, even the best-defended models on ImageNet would drop to nearly 0% accuracy when faced with a strong "white-box" attack (where the attacker knows the model's parameters). This paper investigates whether robust training can finally work at the scale of 1.2 million high-resolution images.
Methods
The authors propose two primary flavors of a technique called Logit Pairing:
- Adversarial Logit Pairing (ALP): During training, the model is shown a clean image and its adversarial counterpart. The loss function is modified to penalize the $L_2$ distance between the logits (the raw output vector before the final softmax) of these two images. This forces the model to develop internal representations that are "smooth" and ignore the noisy perturbations added by an attacker.
- Clean Logit Pairing (CLP): Surprisingly, simply forcing the logits of two randomly selected clean images to be similar also improved robustness, albeit less than ALP. This is a much cheaper "squeezing" technique that doesn't require generating adversarial examples during training.
Additionally, the authors implemented Mixed-minibatch PGD (M-PGD), a scaled-up version of adversarial training that uses a mixture of clean and adversarial examples in every training step.
Key Results
ImageNet White-Box Breakthrough
On ImageNet, previous defenses were crushed by PGD attacks, resulting in approximately 1.5% accuracy. ALP boosted this to 27.9% (Top-1) and 55.4% (Top-5), representing a significant leap in the feasibility of robust large-scale models.
Performance on Standard Benchmarks
Beyond ImageNet, ALP demonstrated dominance on smaller datasets. On MNIST, ALP reached 96.4% accuracy under white-box attacks, surpassing the previous benchmark of 93.2% set by standard adversarial training.
Black-Box Robustness
ALP achieved 47.1% accuracy against black-box attacks on ImageNet, effectively tying the previous state-of-the-art (Tramèr et al., 2018). Furthermore, the authors showed that attacks generated using ALP-trained models could significantly degrade the performance of other existing defenses, suggesting that ALP learns a more "universal" understanding of the adversarial space.
Discussion and Future Work
ALP acts as a powerful regularizer by enforcing a "prior" of smoothness. By forcing the model to treat clean and adversarial versions of an image similarly, it pushes the network to focus on meaningful features rather than high-frequency noise.
However, a "robustness tax" remains: ALP-trained models saw a slight dip in accuracy on clean images (dropping to around 72% on ImageNet). While ALP is empirically strong, it is not a "certified" defense, meaning more sophisticated future attacks could potentially circumvent it. Future research may explore "Feature Pairing"—matching intermediate layer features rather than just final logits—to provide even deeper regularization.
Summary updated