Depth-Truncated Adversarial Training: Robust Where the Taxonomy Says It Matters

Adversarial training makes an image classifier resistant to small, malicious input perturbations by teaching it to give the same answer on an image and on its worst-case perturbed copy. It has a well-known cost: accuracy on ordinary, unperturbed images drops. Current methods demand the same invariance for every distinction the classifier makes. Telling a husky from a malamute and telling a husky from a fire truck both have to survive the perturbation. Yet the second distinction is the one whose failure is dangerous, and the first is the one that depends on fine, low-amplitude visual detail that robustness training tends to erase. We propose to spend the robustness budget unevenly, following a class hierarchy such as WordNet or a biological taxonomy. The model must stay invariant to perturbations at the coarse levels of the tree and may stay sensitive at the leaves.

The mechanism is a small change to the standard TRADES objective. TRADES penalizes the divergence between the model's predictions on clean and perturbed inputs. If the prediction is written as a product of conditional choices down the tree, from root to super-family to family to species, that divergence splits exactly into one term per tree level. We weight these terms by depth and recover TRADES when all weights equal one. The result is a single-backbone training method with no extra networks. It replaces one global robustness knob with a robustness profile over the taxonomy. The question it lets us answer is concrete: at a fixed level of coarse-category robustness, how much clean fine-grained accuracy can be recovered?

Core Claim

Much of the clean accuracy that adversarial training destroys is spent buying invariance at the deep levels of the class tree, where it is least valuable. We claim that removing the deep-level invariance requirement recovers substantial clean fine-grained accuracy while keeping coarse-level robustness at the level of full TRADES. That robustness should hold even under attacks built specifically to push predictions across super-categories. We further claim that this gain comes from the real taxonomy and is not just a side effect of weaker regularization. A shape-matched random hierarchy should not reproduce it.

The Objective

Let the classifier output leaf probabilities $p(y\mid x)$. Internal-node probabilities are sums over descendant leaves. Let $a_d$ denote the ancestor of the label at depth $d$ of the tree, for $d = 1,\dots,D$. The leaf distribution then factors exactly as $p(y\mid x)=\prod_{d} p(a_d\mid a_{<d},x)$. By the chain rule for the Kullback–Leibler divergence (KL), the TRADES boundary term between a clean input $x$ and a perturbed input $x'$ decomposes into per-level conditional divergences. We train with

$$ \min_f \; \mathbb{E}\Big[\mathrm{CE}(f(x),y) + \lambda \max_{x'\in\mathbb{B}(x,\epsilon)} \sum_{d=1}^{D} w_d \,\mathbb{E}_{a_{<d}\sim p(\cdot\mid x)}\,\mathrm{KL}\big(p(a_d\mid a_{<d},x)\,\|\,p(a_d\mid a_{<d},x')\big)\Big], $$

where $\mathrm{CE}$ is cross-entropy on the fine label, $\mathbb{B}(x,\epsilon)$ is the perturbation ball, $\lambda$ is the TRADES trade-off weight, and $w_d\in[0,1]$ is the invariance weight at depth $d$. Setting $w_d=1$ for all $d$ is exactly TRADES. Setting $w_d=1$ only for $d\le k$ asks for robustness down to level $k$ and leaves finer distinctions free. The inner maximization optimizes the same weighted sum, so the attack used during training is itself aimed at the protected levels.

Theoretically Principled Trade-off between Robustness and Accuracy supplies the method we generalize. Its decomposition of robust error into natural error plus boundary error is the lens we refine: we target the boundary error at depth $k$ and leave the leaf-level boundary error alone. The leaf logits, the inference cost, and the architecture are all unchanged. Only the training loss changes.

Relation to Prior Work

Two papers motivate the proposal. Improving Hierarchical Adversarial Robustness of Deep Neural Networks argued that cross-super-category adversarial errors are the dangerous ones. It showed that a worst-case targeted attack drives undefended coarse accuracy to near 0%. Its fix, called HAR, uses one coarse network plus one fine network per super-class and is limited to two-level trees. Its gain on full CIFAR-100 under PGD-200 was small (24.40% vs. 24.00%). We keep its threat model and its targeted attack but replace the multi-network design with a loss on one backbone that works for trees of any depth. DAFA: Distance-Aware Fair Adversarial Training showed that adversarial training hurts most the classes surrounded by similar neighbours. It responds by re-weighting classes and widening margins between close classes, with closeness estimated from softmax outputs. We do almost the opposite: we relax invariance between close (sibling) classes rather than strengthening it. We also take closeness from a fixed taxonomy rather than from noisy predictions.

Three lines of work attack clean-accuracy loss without a taxonomy. Topology-preserving Adversarial Training for Alleviating Natural Accuracy Degradation (TRAIN) trains a second, standard model and makes the robust model copy its neighbourhood graph. Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning (RiFT) fine-tunes the layers least important for robustness on clean data and interpolates weights afterwards. Plain tuning of TRADES's $\lambda$ is the third. All three treat every class boundary alike. We change which boundaries are protected, and we expect our method to be complementary to RiFT as a post-hoc step.

From hierarchical learning we borrow the factorization idea of Making Better Mistakes: Leveraging Class Hierarchies With Deep Networks. That paper's hierarchical cross-entropy (HXE) weights the clean classification loss by depth and was only studied under clean training. We weight the robustness term by depth instead, which is a different place in the objective. Learning Structured Representations by Embedding Class Hierarchy adds a regularizer that makes feature distances follow tree distances. It warns that rigid tree constraints can crowd sub-classes together, which bears on our design: our method imposes no feature-space geometry and constrains only output invariance. LCA-on-the-Line: Benchmarking Out of Distribution Generalization with Class Taxonomies uses tree distance between predicted and true classes as an evaluation tool. We adopt it as a secondary metric.

Research Program

We first build the depth-weighted loss as a drop-in replacement for the TRADES loss. A natural home is the microsoft/robustlearn codebase, which also hosts RiFT, or alibaba/easyrobust. Internal-node probabilities come from summing leaf softmax outputs, so the per-level conditional KLs cost a few sparse matrix products.

Second, we map the robustness profile. We sweep the truncation depth $k$ and $\lambda$ and measure, for each trained model, clean fine accuracy and robust accuracy at every tree level. The deliverable is a set of Pareto frontiers of clean fine accuracy against depth-$k$ robust accuracy, compared with the frontier traced by global $\lambda$ in TRADES.

Third, we test the mechanism. If deep-level invariance is the expensive part, then the clean-accuracy gain should be concentrated on classes with many close siblings, the same classes DAFA identifies as hardest. The gain should also disappear when the real tree is replaced with a random tree of identical shape.

Finally, we report the cost honestly. Fine-level robust accuracy will fall below TRADES by design. The contribution is only worthwhile if coarse-level robustness, measured with the targeted cross-super-category attack, is preserved.

Baselines

The naive reference points are standard training and TRADES at its default $\lambda$. On CIFAR-10, the TRADES paper reports a tunable range up to 88.64% natural accuracy and 56.61% robust accuracy under PGD. These two models bound the clean and robust ends of the comparison. We reproduce them ourselves, because the file reports no TRADES numbers on tieredImageNet-H or iNaturalist-H. No reported result was found on Lacuna for adversarial training on either hierarchical ImageNet benchmark.

The strongest single competitor for recovering clean accuracy is TRAIN layered on TRADES. It reports up to an 8.78% gain in natural accuracy on CIFAR-100, with up to 3.04% more robust accuracy. It is the strongest because it already recovers large clean gains with a one-backbone deployment, even though it trains two networks. We run it with its released recipe, the same backbone, the same attack strength, and the same number of epochs. We also give it the same hyperparameter search budget as ours, counted in trained models.

DAFA is the strongest distance-aware competitor. On CIFAR-10 it reports worst-class robust accuracy rising from 21.31% to 30.53% while average robust accuracy stays near 49%. It is run identically. We also include a DAFA variant that takes class distances from the tree instead of softmax outputs. This isolates whether the gain comes from the taxonomic signal or from our direction of relaxation.

RiFT is run both as a competitor and stacked on our model. HAR is run on CIFAR-100 only, where its 21 networks are affordable; its reported full-CIFAR-100 PGD-200 figure is 24.40%.

Two ablations isolate the key ingredient. The first trains with the same depth weights over a random tree that has the same branching profile. The second is a TRADES model whose global $\lambda$ is tuned to match our coarse-level robustness.

Beating the baselines means the following. At matched depth-$k$ robust accuracy under the targeted cross-super-category attack, our method must reach higher clean fine accuracy than TRADES at any $\lambda$, than TRAIN, and than DAFA. The margin must exceed seed-to-seed variation across three seeds, and the random-tree ablation must not close the gap.

Why not just…?

Why not just lower TRADES's $\lambda$? Lowering $\lambda$ relaxes every boundary at once, so coarse robustness drops together with fine robustness. Ma et al. report that coarse accuracy collapses under targeted attack when boundaries are left unprotected. This alternative is fully settled by the frontier comparison. If the global-$\lambda$ frontier matches or dominates the depth-truncated frontier when measured on depth-$k$ robust accuracy, our claim is false.

Why not just train a robust coarse head and a clean fine head on one shared backbone? This is a real threat. For a two-level tree it is close to our method with weights (1, 0). The differences are that our conditionals come from a single leaf distribution, so coarse and fine predictions cannot contradict each other, and that our weights can be graded smoothly over 8 to 13 levels. We include the two-head model as a baseline on all datasets. If it matches us on tieredImageNet-H and iNaturalist-H, where a graded profile should matter most, we would report the simpler model as the recommended method.

Why not just apply RiFT-style fine-tuning and interpolation to a flat TRADES model? RiFT reports clean gains of 1.5–2.0% on Tiny-ImageNet. However, it can only reshape features the robust model already learned, and it does not know which robustness it is giving up. We test RiFT alone, our method alone, and both together. If our method gives larger clean gains at matched coarse robustness, or if the two add up, the proposal holds.

Why not just use DAFA or TRAIN? Both are strong. Neither distinguishes a dangerous error from a benign one: TRAIN copies an empirical neighbourhood graph, and DAFA widens margins between exactly the sibling classes we propose to relax. The tree-distance DAFA variant tells us whether the taxonomy merely improves their distance estimates or whether the direction of relaxation is what matters.

Proposed Experiments and Decisive Checks

We use three benchmarks. CIFAR-100 provides 100 fine classes under 20 super-classes, with ResNet and WideResNet backbones. tieredImageNet-H has 608 classes on a 13-level WordNet tree. iNaturalist-H has 1,010 species on an 8-level taxonomy. Both ImageNet-scale sets are downsampled to 64×64 to keep 10-step projected gradient descent (PGD) training affordable. Their clean reference points come from Bertinetto et al., where HXE reached 27.68% top-1 error against 27.55% for standard training on tieredImageNet-H.

Attacks include untargeted PGD at increasing step counts. They also include the worst-case targeted attack of Ma et al., generalized to any depth $k$: it iterates over leaves outside the true depth-$k$ ancestor and takes the most damaging target.

Metrics are clean fine top-1 accuracy, robust accuracy at each depth, the average tree distance of mistakes on clean and attacked inputs, and worst-class clean and robust accuracy.

Three checks decide the claim. The first is whether, at matched coarse robust accuracy under the targeted attack, depth-truncated training beats the TRADES global-$\lambda$ frontier on clean fine accuracy on all three datasets. The second is whether the random-tree ablation loses most of that gain; if it does not, the effect is regularization strength, not taxonomy. The third is whether the advantage grows with tree depth, from CIFAR-100 to iNaturalist-H to tieredImageNet-H, as the share of invariance spent on deep levels grows.

A positive answer on all three would show that a substantial part of the robustness–accuracy trade-off is a choice about which distinctions to protect, not a fixed tax. A negative answer on the second check would be equally informative, because it would mean class hierarchies add nothing beyond a weaker TRADES.