Adversarial training makes an image classifier resistant to small, deliberately crafted pixel changes. It does this by training on the worst such changes it can find. The known cost is that accuracy on ordinary, unperturbed images drops. One recent explanation holds that part of this drop comes from adversarial perturbations that push images "off" the set of natural-looking images. Training on those perturbations bends the decision boundary in ways that clean images later pay for. TART acts on this idea. It estimates the local directions of natural variation with an autoencoder and principal component analysis, keeps adversarial examples whose perturbation mostly follows those directions, and trains on the clean image otherwise. The method raises clean accuracy, but its authors say it cannot run alongside online data augmentation. Every randomly cropped, flipped or shifted image would need its own tangent computation, and the tangents for the whole dataset must be precomputed and stored. This is a serious limitation, because augmentation is where most of the recent gains in adversarial training come from (Data augmentation alone can improve adversarial training, Data Augmentation Can Improve Robustness).
We propose to drop the learned manifold. We compute tangent directions in closed form from the geometric transformations themselves. A small rotation, shift, scale, shear or smooth local warp of an image changes each pixel by the image gradient dotted with a known displacement field. The directions along which an image varies under these transformations are therefore available analytically, for any augmented image, at the cost of one finite-difference gradient. With these directions, tangent-guided adversarial training becomes compatible with strong geometric augmentation, needs no storage, and runs on any dataset. It also yields a sharp test of whether the "on-manifold" story explains the clean-accuracy gains at all, or whether TART-style methods mostly work because they train on clean images part of the time.
Core Claim
The claim is testable. On CIFAR-10, CIFAR-100, SVHN and Tiny ImageNet, adversarial training that scales each example's perturbation budget by how much of the perturbation lies in a closed-form geometric tangent space should do three things:
- Recover at least TART's clean-accuracy gain over plain adversarial training, without an autoencoder.
- Stack with the diversity-focused geometric augmentation pipeline of IDBH and with weight averaging, which TART cannot do.
- Beat two controls: a matched random-clean-mixing control and a random subspace of the same dimension.
If the controls match the method, then geometry is not the mechanism, and we will report that as the main finding.
Mechanism
Consider an image $x$ as a function of pixel coordinates $u$, and a displacement field $v(u)$. The infinitesimal effect of transforming the image along $v$ is the tangent vector
$$t_v(x)(u) = -\nabla_u x(u) \cdot v(u),$$
where $\nabla_u x$ is the spatial image gradient, computed per channel. The six affine generators give translations ($v$ constant), rotation ($v(u) = (-u_2, u_1)$), isotropic and anisotropic scaling, and shear. To these we add $K$ smooth, low-frequency displacement fields (for example, low-order sinusoidal bases). These capture the local elastic variation that affine maps miss, and they raise the tangent dimension from 6 to a tunable $d = 6 + K$. After orthonormalizing the $d$ vectors, we measure, for the adversarial perturbation $\delta$ found by PGD (projected gradient descent, the standard iterative attack), the tangential fraction
$$\rho(\delta) = \frac{\lVert P_T \delta \rVert_2^2}{\lVert \delta \rVert_2^2},$$
where $P_T$ projects onto the span of the tangent vectors at the current, already-augmented image.
Instead of TART's hard rule (full budget or zero), we set each example's budget as $\epsilon_i = \epsilon \cdot s(\text{rank}(\rho_i))$. Here $s$ is a monotone map on the within-batch rank, and the budget of the final attack is re-run at $\epsilon_i$. The hard 0/$\epsilon$ rule is kept as a special case.
The extra cost is one gradient image and a $d \times 3072$ Gram–Schmidt step per image (for $32 \times 32$ inputs), which is negligible next to multi-step PGD. This contrasts with the roughly two-fold overhead reported for saliency-guided PART.
Relation to Prior Work
Three papers supply the method.
-
TART supplies the selection idea. We replace its learned, offline tangent estimate with an analytic, online one. - Tangent-Normal Adversarial Regularization splits perturbations into tangent and normal parts using VAE or GAN manifold estimates, but only for semi-supervised clean error. It does not use analytic transformation generators and does not evaluate robustness to attack.
-
Invariance-inducing regularization motivates why geometric directions are the right tangent set. It proves that under label-invariant bijective transformations, spatial robustness and clean accuracy do not conflict in the infinite-data limit. Its target, however, is worst-case spatial transformations, not additive pixel attacks.
Two papers constrain the design.
- Can we have it all? shows that pushing rotational invariance, through augmentation or group-equivariant networks, degrades robustness to $\ell_\infty$ attacks nearly linearly. A 180° rotation range cuts PGD robustness on CIFAR-10 by more than half. We therefore use infinitesimal generators only to measure perturbations. We do not train on large rotations, and we report spatial robustness so that any hidden trade-off is visible.
- The "hardness trap" of IDBH warns that overly hard augmentation collapses both accuracies. We therefore reuse IDBH's calibrated geometric pipeline rather than inventing a harder one.
Rethinking data augmentation for adversarial robustness finds that classical flips and rotations drive most of the decision-surface smoothing credited to newer augmentations. This is further motivation that the geometric subspace, rather than an arbitrary learned one, carries the signal.
We deliberately avoid equivariant architectures. The robustness claims made for them in Bridging Symmetry and Robustness conflict with Can we have it all? and have not been checked with AutoAttack. Resolving that conflict is a separate project.
Research Program
The first stage is diagnostic and runs before any new training. On unaugmented CIFAR-10 we compute $\rho$ under both the analytic tangents and a reimplemented TART autoencoder tangent space, and ask whether the two rankings agree. We then measure the typical size of $\rho$ as $d$ grows. A 6-dimensional subspace of a 3072-dimensional space will capture little of an $\ell_\infty$ perturbation's energy. The question is whether the ranking across examples is still informative, measured as rank correlation against a random $d$-dimensional subspace.
Most importantly, we test the causal premise directly. Starting from a fixed checkpoint, we fine-tune briefly on batches of adversarial examples filtered to high versus low $\rho$, and measure the change in clean test accuracy. If low-$\rho$ perturbations do not hurt clean accuracy more than high-$\rho$ ones, the premise fails and the project pivots to reporting that.
The second stage is training. We train PreAct ResNet-18 and WideResNet-34-10 with standard adversarial training and TRADES, under three settings: plain crop-and-flip, IDBH augmentation (starting from the AROID codebase where it matches the setup), and IDBH plus weight averaging. Each setting is run with and without our budget rule.
The third stage is analysis. We sweep $d$ to see whether local elastic fields add signal beyond the affine ones. We compare hard and soft budget maps. We also track robust overfitting curves, since rescaling budgets could change when robustness peaks.
Baselines
The naive reference point is standard PGD adversarial training at $\epsilon = 8/255$ with random crop and flip, at the same architecture, epochs and learning-rate schedule. The reported numbers vary with setup. Understanding and Mitigating the Tradeoff reports 87.3% clean and 45.8% robust accuracy. TART reports 81.76% clean and 35.89% AutoAttack accuracy on WideResNet-32-10. Because of this spread, we rerun it ourselves rather than borrowing numbers.
The most direct competitor is TART itself, run on unaugmented data as its authors intend. Its reported result is 83.73% clean and 36.58% AutoAttack on CIFAR-10 with WideResNet-32-10, and 84.31% clean when combined with MART. Matching TART's clean gain without its autoencoder is the floor. Beating it requires exceeding its clean accuracy at equal or better AutoAttack accuracy, in our rerun, on the same architecture.
The strongest augmentation-based competitor is IDBH. It reports 50.74% AutoAttack versus 48.21% for the crop-and-flip baseline on PreAct ResNet-18, and 58.16% with weight averaging on WideResNet-34-10. Beating it means a clean-accuracy gain of at least one point with AutoAttack accuracy within noise (three seeds) or better. That gain must hold at the IDBH-plus-weight-averaging operating point, which is where TART cannot run.
We also report CutMix with weight averaging from Data Augmentation Can Improve Robustness, at 60.07% AutoAttack on WideResNet-28-10 CIFAR-10, as a reference frontier. TRAIN serves as a clean-accuracy-focused competitor, with gains of up to 8.78% natural accuracy reported on CIFAR-100 with TRADES and no absolute value in the file.
Two ablations isolate the key ingredient.
- Matched clean-fraction control. Each example gets the same distribution of budgets our rule produces, but assigned at random rather than by $\rho$. This separates "geometry picks which examples" from "some examples are simply attacked less."
- Random-subspace control. $\rho$ is computed against a random orthonormal subspace of the same dimension $d$, recomputed per image.
All methods share data, augmentation, epochs, attack steps and a matched hyperparameter search budget of equal trial count. AutoAttack from RobustBench is the robustness metric throughout.
Why not just…?
Why not just train on clean images some fraction of the time? This is the real threat. TART's hard rule sends roughly half the batch to $\epsilon = 0$, and mixing clean and adversarial examples is a known way to buy clean accuracy with robustness. The file contains no evidence that TART beats such a control, because TART does not report one. Our matched clean-fraction ablation settles it directly. If it reaches the same point on the clean-versus-robust curve as the $\rho$-guided rule, we will conclude that tangent selection adds nothing and say so.
Why not just use IDBH with weight averaging and stop? It is the strongest augmentation-only recipe in the file (58.16% AutoAttack on WideResNet-34-10). The weight-averaging work shows that augmentation and averaging stack, but neither addresses the clean-accuracy gap directly. We run our rule on top of exactly this recipe. If it gives no clean gain there, the method has no practical value, whatever it does on bare adversarial training.
Why not just add external unlabeled data with robust self-training? It is the most effective known lever: 91.8% clean and 58.5% robust on CIFAR-10 in Understanding and Mitigating the Tradeoff. But it needs 500K related unlabeled images and a good pseudo-labeler, and it still leaves a gap to standard training. We treat it as an upper reference, not a competitor. We also test whether our rule adds to it on CIFAR-10, where the external data is available.
Why not just run TART on precomputed tangents of augmented copies? One could store tangents for a fixed set of augmented versions of each image, or compute tangents on the original image and reuse them after augmentation. The first multiplies TART's storage problem and caps augmentation diversity, which IDBH shows is what matters. The second applies tangents of one image to a different, shifted image. We implement the reuse variant as a baseline under IDBH to measure how much the mismatch costs.
Proposed Experiments and Decisive Checks
The benchmarks are CIFAR-10 and CIFAR-100 (primary), SVHN (natural tilt and jitter, where IDBH reports a 13.23% robustness gain and CutMix is known to fail), and Tiny ImageNet (scale, where TART was never tested). CIFAR-10-C measures corruption robustness. We report four metrics:
- clean accuracy,
- AutoAttack accuracy at $\ell_\infty$, $\epsilon = 8/255$,
- worst-of-10 spatial accuracy over ±30° rotations and ±3-pixel shifts, following the invariance-regularization protocol, to catch the spatial-versus-$\ell_\infty$ trade-off,
- per-epoch wall-clock overhead.
All results use three seeds with confidence intervals.
Three outcomes decide the project:
- In the stage-one causal test, low-$\rho$ adversarial batches must hurt clean accuracy measurably more than high-$\rho$ batches. The analytic ranking must also agree with the autoencoder ranking better than a random subspace does.
- The $\rho$-guided rule must beat both the matched clean-fraction and random-subspace controls on the clean-versus-AutoAttack curve.
- The clean gain must survive under IDBH with weight averaging, with overhead under 10% and no loss in spatial accuracy.
Failing the first means the on-manifold explanation of clean degradation does not hold for geometric tangents. Failing the second means TART-style gains are a budget effect, not a geometric one. Failing only the third means the idea is correct but superseded by augmentation.
References
- TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training
- Data augmentation alone can improve adversarial training
- Data Augmentation Can Improve Robustness
- Improving Accuracy-robustness Trade-off via Pixel Reweighted Adversarial Training
- Tangent-Normal Adversarial Regularization for Semi-supervised Learning
- Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness
- Can we have it all? On the Trade-off between Spatial and Adversarial Robustness of Neural Networks
- Rethinking data augmentation for adversarial robustness
- Bridging Symmetry and Robustness: On the Role of Equivariance in Enhancing Adversarial Robustness
- Understanding and Mitigating the Tradeoff between Robustness and Accuracy
- Topology-preserving Adversarial Training for Alleviating Natural Accuracy Degradation
- RobustBench: a standardized adversarial robustness benchmark