Keeping the Shape of Clean Features During Adversarial Training

Adversarial training, which means training a classifier on deliberately worst-case perturbed inputs, buys robustness at a steady price in ordinary accuracy. On CIFAR-10, standard adversarial training reaches about 87% clean accuracy, while a naturally trained network exceeds 95%. This proposal tests one specific explanation for where that accuracy goes. As the robust model learns, the connectivity of its clean-image features gets damaged: classes that a normally trained network keeps as well-separated clusters start merging at small distance scales, and single classes split into fragments. The size, stretch and orientation of those clusters, by contrast, do not need to be preserved for accuracy.

If this is right, the fix is narrow. While training the robust model, we add a penalty that keeps the clean-feature cluster structure of a frozen, naturally trained network, and we leave its exact distances free. The penalty is a scalable topological comparison of two point clouds that looks only at connected components and the scales at which they merge. Because the constraint is weaker than copying the teacher's features, it should recover clean accuracy without inheriting the teacher's fragility. The project first checks whether the diagnosis is true, and only then whether the regularizer helps.

Core Claim

The claim has two parts. The first is a diagnostic: the clean-accuracy gap of an adversarially trained model is predicted by a topological discrepancy between its clean features and a natural model's clean features. The measure is RTD-Lite from RTD-Lite, which compares the minimum spanning trees (the shortest set of edges connecting all points) of two weighted graphs over the same points, and so captures how clusters merge across scales. We claim this measure predicts the gap better than pointwise feature distance or kernel similarity does.

The second part is a method. With $f_\theta$ the robust model being trained, $g$ a frozen naturally trained teacher, $\phi(\cdot)$ penultimate-layer features, and $\mathcal{L}_{\text{TRADES}}$ the standard TRADES objective, we train with

$$\mathcal{L} = \mathcal{L}_{\text{TRADES}}(\theta) + \mu \cdot \text{RTD-Lite}\big(\bar D_{\phi_g}(B),\, \bar D_{\phi_\theta}(B)\big).$$

Here $B$ is a mini-batch of clean images, $\bar D$ is the pairwise feature-distance matrix divided by its median so that global scale is free, and $\mu$ sets the strength. We claim this moves the clean-versus-robust Pareto front outward compared with TRADES alone and with pointwise teacher distillation.

The penalty never touches adversarial inputs or the test-time computation. That matters given the feature-scattering lesson discussed below.

Positioning

Topology has entered adversarial robustness only at the edges of the pipeline. TopoReformer purifies OCR inputs through a topological autoencoder before they reach the classifier. It keeps clean performance on MNIST, but its F1 score falls to 64.71% under the BPDA adaptive attack, and its authors name end-to-end topology-aware training as future work. Topological Signatures of Adversaries in Multimodal Alignments shows that attacks scatter CLIP and BLIP embeddings and raise their total persistence, and uses this only to detect attacks. Both papers motivate the proposal by finding topological signal around adversarial examples. We differ by applying topology inside training, and to clean features rather than adversarial ones.

Feature Scattering is the closest training-time precedent at the batch level. It deliberately maximizes an optimal-transport distance between clean and perturbed features, and later work on obfuscated gradients warns that defenses of this kind can look robust only because they hide useful gradients from the attacker. Its high PGD-20 number of 70.5% therefore constrains our design rather than inspiring it. Our regularizer pulls toward clean structure, never pushes features apart, and adds nothing at inference.

TART is the strongest geometric explanation of the gap. It argues that perturbations pointing off the data manifold warp decision boundaries, and it keeps only perturbations that move along locally estimated tangent directions. TART changes which perturbations are used. We change what the representation must keep, and we use a global, multi-scale invariant where TART uses local tangent planes.

TRADES and Robust Self-Training frame the trade-off in two ways: as an inherent tension between natural and boundary error, or as a finite-data generalization failure. Our diagnostic gives a concrete way to probe the second view, because it asks whether accuracy loss tracks a measurable structural defect that more data, or a structural prior, could repair.

The topological tools are reused, not invented. RTD-Lite supplies a sub-differentiable comparison that costs $O(n^2)$ in the number of points, so batches of 512 are cheap. Learning topology-preserving data representations proves that a zero divergence of this family means identical persistence barcodes, the record of when each topological feature appears and disappears across scales. Towards Scalable Topological Regularizers supplies a smoother kernel-based variant that also covers loops, which we use as an ablation.

Research Program

The first phase is purely diagnostic and could end the project. We will take public-protocol training runs of standard adversarial training and TRADES on CIFAR-10 and CIFAR-100, saving checkpoints throughout training, together with a naturally trained network of the same architecture. For each checkpoint we compute four discrepancies between student and teacher clean features on held-out batches: RTD-Lite, linear kernel alignment (CKA), maximum mean discrepancy, and mean pointwise feature distance after scale normalization. We then ask which one best predicts the clean-accuracy gap, both across checkpoints and per class, where the per-class version tests whether classes whose clusters merged early in the student's spanning tree are the ones losing accuracy.

The test is pre-registered. If RTD-Lite does not beat pointwise distance and kernel similarity as a predictor, the topological framing is unjustified and we stop. RTD-Lite's monotone response to cluster and ring counts, where CKA behaves erratically on the rings benchmark in RTD-Lite, is the reason to expect it to pass.

The second phase builds the regularizer above on top of TRADES and on top of standard adversarial training. We include a directed variant that penalizes only merges present in the student but absent in the teacher, following the directional idea of Topological Autoencoders. We also include a variant that computes the penalty on unlabeled images, which tests whether a structural prior can partly substitute for the 500,000 extra images that Robust Self-Training uses.

The third phase is a controlled synthetic study. We turn the SPHERES benchmark into a classification task: each nested sphere is a class, embedded in high dimension. This setting makes local tangent estimates unreliable while global connectivity stays well defined, so it lets us see whether a topological constraint helps exactly where TART's local linearization should struggle.

Baselines

The naive reference points are a naturally trained network, which gives the clean-accuracy ceiling and zero robustness, and standard adversarial training with PGD. On WideResNet-32-10, TART reports standard adversarial training at 81.76% clean and 35.89% under AutoAttack, the standard ensemble of strong white-box and black-box attacks.

TRADES is the main host and the main competitor, because it already exposes a tunable clean-versus-robust curve. Its paper reports 88.64% clean accuracy and 56.61% PGD robust accuracy on CIFAR-10, but the PGD number is not directly comparable to AutoAttack. We will therefore rerun TRADES ourselves across at least five values of its trade-off weight and compare full Pareto curves under AutoAttack.

The strongest existing method aimed squarely at clean accuracy is TART, reported at 83.73% clean and 36.58% AutoAttack on WideResNet-32-10, and 84.31% clean when combined with MART. We will run TART and our method on the same architecture, with the same epoch budget, learning-rate schedule and PGD-10 inner loop. Each method gets the same number of hyperparameter trials. We also report wall-clock time, since TART costs 1.24 times standard training. Beating TART means higher clean accuracy at equal or higher AutoAttack accuracy, averaged over three seeds, by more than the seed standard deviation.

Robust Self-Training is a data-rich reference only: it reports 91.8% clean and 58.5% robust with 500,000 unlabeled images. We compare against it only in the unlabeled-data variant, at matched unlabeled data size.

The ablation that isolates the key ingredient keeps everything else fixed and replaces the RTD-Lite term with alternatives that use the same teacher and the same $\mu$ sweep. The alternatives are pointwise feature distillation (mean squared error on scale-normalized features), logit distillation on clean inputs, CKA matching, and the loop-aware PPM-Reg term. Our method only matters if it beats pointwise and CKA matching.

Why not just…?

Why not just distill the clean teacher's features or logits pointwise? This is the real threat, and we treat it as a primary comparison, not a strawman. The argument against it is that pointwise matching asks the robust model to reproduce the teacher's exact feature distances, including the directions along which the teacher is fragile. That should pull the solution back toward non-robust geometry, so clean accuracy rises but AutoAttack accuracy falls. A topological match constrains only which points connect at which relative scale. The phase-two ablation measures this directly, at matched $\mu$ and compute. If pointwise distillation reaches the same Pareto front, we will report that the topological ingredient is unnecessary.

Why not just tune the TRADES weight? Tuning the TRADES weight slides along one fixed curve. It cannot move the curve itself. Our claim is that the curve shifts outward, which the full sweep against TRADES checks directly. A single tuned operating point would not count as evidence.

Why not just use TART's tangent-space filtering? TART and our method act on different objects, perturbations versus representations, so they can be stacked. We run TART alone, our method alone, and both together. The SPHERES study tests the specific case where local tangents should fail. If TART alone matches the combination on CIFAR-10, the topological term adds nothing on natural images, and we will say so.

Why not just add more unlabeled data, as Robust Self-Training does? When large unlabeled pools exist, they are the simplest fix. Our unlabeled-data variant asks a narrower question: at 10,000 to 50,000 unlabeled images, does the structural penalty recover a meaningful share of what Robust Self-Training gains at 500,000? This is motivated by PPM-Reg's jump from 48% to 75% with 200 labels on Kuzushiji-MNIST.

Proposed Experiments

The decisive experiments run on CIFAR-10 and CIFAR-100 under the $\ell_\infty$ threat model with $\epsilon = 8/255$, using WideResNet-32-10 for comparability with TART and ResNet-18 for the large sweeps. We report clean accuracy and AutoAttack accuracy as full Pareto curves, plus per-class clean accuracy.

The first decisive check is the phase-one correlation test, with its pre-registered stopping rule. The second is the Pareto comparison against TRADES, TART and pointwise distillation under matched budgets.

The third is a gradient-masking audit that follows the checklist of Obfuscated Gradients. It checks that the black-box Square attack inside AutoAttack does not beat the white-box attacks, that robust accuracy falls toward zero as $\epsilon$ grows, and that adding attack iterations does not change the conclusions. Because the regularizer touches only training-time clean features, we expect this audit to pass, and a failure would discredit the method.

Finally, the SPHERES classification study and the reduced-unlabeled-data study test, respectively, whether global structure helps where local geometry fails and whether a structural prior can stand in for scale.