Several papers report that training a classifier on "on-manifold" adversarial examples, meaning realistic variations such as changed lighting, pose or texture rather than pixel noise, makes the classifier more robust without costing clean accuracy, and sometimes improves it. That claim stands against a well-known result: hardening a model against worst-case perturbations usually lowers its accuracy on ordinary inputs. There is a catch in how the claim is tested. "The data manifold" is never known. It is approximated by a generative model, and the on-manifold robustness of the trained model is then measured with attacks built from that same generative model. A classifier trained against one generator's realistic edits may simply have learned that generator's particular modes of variation.
We propose to measure what such training buys when the attacker uses a different generator, or the true manifold where it is known. In two sentences: train classifiers with on-manifold adversarial examples from generator A, then attack them adaptively with generators B and C and with the exact data manifold on synthetic tasks where it exists. Use the results to test whether the clean-accuracy gains survive, and whether the robustness that transfers is predicted by how well A's local directions of variation line up with the true ones.
Core Claim
Our hypothesis is that the "no trade-off" result is partly an artifact of evaluating with the training generator. When on-manifold robustness is measured by an attacker whose manifold model differs from the defender's, we expect three things. The robustness gain should shrink. A robustness-accuracy trade-off should reappear along the curve traced by the training strength. And the size of the gain that does transfer should be predicted by the overlap between the defender's estimated tangent space and the attacker's (or the true) tangent space. The tangent space here is the set of small directions in which an image can change and stay realistic.
Concretely, for an input $x$, let $P_{\hat T}$ project onto the $k$-dimensional tangent space implied by the defender's generator and $P_{T}$ onto the reference tangent space. We define the alignment score
$$A(x) = \tfrac{1}{k}\,\| P_{T} P_{\hat T} \|_F^2 ,$$
which is 1 when the two subspaces coincide and near 0 when they are orthogonal. The prediction is that the held-out on-manifold robust accuracy gained per point of training strength rises monotonically with the dataset-average $A$. If instead transferred gains are large regardless of alignment, the free-lunch reading is vindicated.
Why This Gap and Not the Obvious One
The obvious project here is "latent-space adversarial training with a TRADES-style regularizer, evaluated with AutoAttack." We think it is the wrong first step, for two reasons.
First, the generator-matched evaluation it would inherit is exactly the weak point. Disentangling Adversarial Robustness and Generalization. trains on on-manifold examples from a VAE-GAN and finds lower test error, concluding that robustness and generalization need not conflict. Robustness and Generalization via Generative Adversarial Training (GAT) does the same with StyleGAN latents on LSUN and reports clean accuracy rising from 88.9% to 89.5%. These two papers motivate this proposal. Neither evaluates on-manifold robustness with an independent generator.
Second, the adaptive-attack gap that is usually proposed for tangent-based training is partly a mirage. TART uses an autoencoder's local tangent estimate only to decide which training examples get full perturbations. At test time it is an ordinary classifier, so AutoAttack is already a fully adaptive pixel attack against it. What is genuinely untested is whether TART-style models are robust to on-manifold attacks at all.
Several papers constrain the design. Understanding adversarial robustness against on-manifold adversarial examples shows that pixel adversarial training hardens low-variance directions and leaves top-variance directions fragile: an MNIST model at 95.51% under regular attacks drops to 15.53% under generative on-manifold attacks. This tells us pixel-trained baselines must be probed on-manifold. Adversarial Spheres and Adversarial vulnerability for any classifier prove that on-manifold adversarial examples are unavoidable in high dimension. The second paper gives a CIFAR-10 in-distribution robustness bound of about 0.10 against an observed 0.008, so we report relative gains at fixed budgets, never "solved" robustness. Theoretically Principled Trade-off between Robustness and Accuracy (TRADES) supplies the decomposition of robust error into natural and boundary error that we use to read the curves.
Research Program
The program has three parts: a label-safe on-manifold attack suite with held-out generators, a controlled ladder of training methods, and the alignment analysis.
The attack suite contains several kinds of on-manifold attacker, all run white-box against the classifier with step-size-adaptive optimization in the spirit of AutoAttack. Their manifold models differ from the defender's.
- A Stable Diffusion VAE latent attacker, following Latent Transfer Attack, whose latent edits concentrate energy in low frequencies.
- A normalizing-flow attacker, following AdvFlow and AFLOW. Flows are exactly invertible, so the starting latent is exact rather than an approximation.
- A generator-free attacker that perturbs only the top principal directions of the data, the Eigen-AE of the "Understanding" paper.
- On synthetic tasks, an attacker that moves along the true manifold.
The defender's own generator is also used as a white-box attacker, so every model has a matched score and a held-out score.
The main methodological risk is identity flipping, which GAT itself flags: large latent moves can change what the image actually shows. We handle it in three ways.
- On synthetic data the true label is a known function of the generating parameters. We use concentric spheres from Adversarial Spheres and a rendered-glyph dataset in the style of FONTS, where the latent parameters, and hence the oracle label and tangent space, are known exactly.
- On CIFAR-10 and CelebA-HQ we calibrate each attacker's budget so that a fixed fraction of attacked samples is confirmed label-preserving by human raters on a random subset. GAT reports 99.2% rater agreement with ground-truth labels for its edits, and we target a comparable rate. Budgets across generators are then matched by validity rate rather than by latent norm, which is not comparable across generators.
- Gradients through decoders and flows are checked for masking by comparing against gradient-free search on a subset. This follows the lesson of DiffBreak and Unlocking The Potential of Adaptive Attacks on Diffusion-Based Purification: broken backpropagation through generative pipelines fakes robustness.
The training ladder holds the architecture fixed. On CIFAR-10 this is WideResNet-32-10, to match TART. The ladder is:
- standard training;
- pixel PGD adversarial training;
- TRADES;
- TART;
- GAT-style latent adversarial training on generator A;
- the same with a TRADES-style consistency term in place of the cross-entropy on adversarial points;
- joint training mixing pixel and latent examples, the conceptual proposal of the "Understanding" paper.
Each latent method is swept over its strength parameter (the latent budget, or the TRADES weight $\lambda$). The result is a full clean-versus-robust curve per method, not a single point.
The alignment analysis computes $A(x)$ for every defender–attacker pair. Tangent spaces come from decoder Jacobians, or from local PCA for the generator-free attacker as in TART, and from the analytic manifold on synthetic data. We then regress held-out robustness gain on alignment.
Baselines
The naive reference point is standard training. It fixes the clean-accuracy ceiling and the floor on every attack, so a method only "breaks the trade-off" if its clean accuracy stays at or above this model's.
The strongest existing method for the clean-accuracy side of the question is TART. On CIFAR-10 with WideResNet-32-10 it reports 83.73% clean accuracy and 36.58% AutoAttack robust accuracy, against 81.76% and 35.89% for standard adversarial training (TART). We will reproduce it with the authors' autoencoder-derived tangent rule, our data pipeline and our epoch budget. Beating it means one of two things. Either a latent method matches TART's clean accuracy within reproduction noise while exceeding its held-out on-manifold robust accuracy at matched validity budgets, or TART's own held-out on-manifold robustness turns out no better than pixel adversarial training's. That second outcome is itself a finding about what tangent guidance buys.
TRADES is the principled trade-off baseline. Its reported numbers are 88.64% clean and 56.61% robust under strong PGD (TRADES). Those predate AutoAttack, so we re-evaluate rather than quote them.
GAT is the strongest existing latent-training method. Its reported numbers are 89.5% clean accuracy on LSUN versus 88.9% for standard training, and 42.1% mean accuracy under unseen attacks versus 13.0% for pixel adversarial training (GAT). We run it on LSUN to reproduce those numbers before extending it.
For Disentangling Adversarial Robustness and Generalization. we found no reported result on Lacuna in numeric form, so its VAE-GAN on-manifold training is rebuilt on the glyph and Fashion-MNIST settings.
The key ablation holds the training method fixed and changes only the evaluator: the defender's own generator versus a held-out one. The gap between those two numbers is the quantity this proposal is about.
All methods get the same data, augmentation, epochs and hyperparameter search budget, counted in GPU-hours. The extra cost of latent methods is reported, not hidden.
Why not just…?
Why not just scale up pixel adversarial training? Larger capacity may raise pixel robustness, but the evidence says pixel training defends the wrong directions. CIFAR-10 accuracy under a top-500-eigenvector attack falls to 12%, versus 22% for the bottom 500 (Understanding adversarial robustness against on-manifold adversarial examples). We include a larger-width pixel-trained model in the ladder. If its held-out on-manifold robustness matches the latent methods, capacity rather than manifold-awareness is doing the work, and we will say so.
Why not just evaluate with the defender's generator, as GAT and the Disentangling paper did? Because that measures whether the model learned one generator's variations, not the data's. The matched-versus-held-out ablation settles this directly. If the two scores agree across generators, generator-matched evaluation is vindicated and the free-lunch claim strengthens.
Why not just use strong standard augmentation such as color jitter, random geometry and learned augmentation policies? This is a real threat. Ordinary augmentation could deliver both the clean-accuracy gain and much of the held-out on-manifold robustness, since many "semantic" edits are simple photometric changes. GAT's comparison was against pixel adversarial training rather than a strong augmentation policy, so the file does not rule this out. We therefore train an augmentation-only model at matched compute and place it on the same curves. Latent adversarial training earns its cost only if it beats this model on held-out robustness at equal clean accuracy.
Why not just purify inputs at test time with a generative model? Purifiers have repeatedly been shown to rely on broken gradients: DiffPure falls from about 62% to 17.19% on CIFAR-10 under exact gradients (DiffBreak). The ELBO-based purifier of Adversarial Purification with the Manifold Hypothesis runs about 102 times slower than a forward pass. Purification also does not address our question, which is what training on realistic edits does to the classifier itself.
Proposed Experiments
The first experiment sits on the synthetic testbeds, where the true manifold is known. We train the ladder on concentric spheres and on rendered glyphs, with defender generators of deliberately varied fidelity. These range from a well-fit flow to a small, misspecified VAE, which gives a controlled spread of alignment scores. We then measure robust accuracy along the true manifold at fixed geodesic radii. The decisive check is the regression of transferred gain on $A$. A positive, monotone relation with matched-evaluator gains far above true-manifold gains supports the core claim. A flat relation with comparable gains refutes it.
The second experiment repeats the ladder on CIFAR-10, and on LSUN to reproduce GAT. Each model is evaluated by AutoAttack in pixel space and by the four on-manifold attackers at validity-matched budgets. Results are plotted as clean-versus-robust curves per method and per evaluator. The decisive pattern is the comparison between curves. Under the defender's own generator, latent methods should show clean accuracy flat or rising with training strength. Under held-out generators, a falling clean-versus-robust curve would mean the trade-off reappears once the evaluator changes. The analysis uses the TRADES decomposition to ask whether the reappearing loss is natural error or boundary error.
The third experiment, conditional on the claim holding, asks whether the gap can be closed cheaply. We train against a randomized mixture of generators A and B and test on held-out C. The prediction is that held-out gains track the alignment of the mixture's combined tangent spaces with C's. As throughout, a result only counts if gradient-free checks on a sample confirm the gradients through each generator were not masked.
References
- Disentangling Adversarial Robustness and Generalization.
- Robustness and Generalization via Generative Adversarial Training
- TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training
- Understanding adversarial robustness against on-manifold adversarial examples
- Adversarial Spheres
- Adversarial vulnerability for any classifier
- Theoretically Principled Trade-off between Robustness and Accuracy
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
- Latent Transfer Attack: Adversarial Examples via Generative Latent Spaces
- AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flows
- AFLOW: Developing Adversarial Examples Under Extremely Noise-Limited Settings
- DiffBreak: Is Diffusion-Based Purification Robust?
- Adversarial Purification with the Manifold Hypothesis