When categories overlap, some errors are nearly harmless and others are serious. Labeling a physician "doctor" or "person" is barely wrong, while labeling them "patient" or "organization" is badly wrong. Adversarial training is the standard defense against small, deliberate input edits such as synonym swaps: it trains the model on the worst-case edit it can find. Yet it treats every flipped prediction as equally bad. This causes two problems in large, overlapping label spaces such as hierarchical topic taxonomies or fine-grained entity types. The defense spends effort resisting flips to near-equivalent labels. It also pushes the model to treat closely related classes the same way, which blurs the fine distinctions the task depends on.
We propose to replace the "any flip" objective with a taxonomy-weighted one. The attacker searches for the edit that moves the prediction as far as possible through the label hierarchy. The defender minimizes that distance. Distance is measured with a tree-structured transport cost that is cheap to compute. The claim to test is that, for the same training budget, this yields models whose mistakes under attack are much less severe, and whose clean accuracy on deep, fine-grained classes suffers less than under standard adversarial training.
Core Claim
Let $T$ be the label hierarchy, with edge weights $w_e$ that shrink with depth. For each edge $e$, let $\Gamma_e$ be the set of labels below it. For a predicted distribution $p$ and a target distribution $q$ over labels, the tree-Wasserstein distance (the minimum cost of moving probability mass from $p$ to $q$ along tree edges) has a closed form:
$$W_T(p,q)=\sum_{e\in T} w_e\,\big|p(\Gamma_e)-q(\Gamma_e)\big|.$$
Under this cost, confusing two deep siblings is cheap, and confusing two top-level branches is expensive. Multi-label targets, such as a full path from root to leaf, are handled by spreading the target mass over the deepest gold labels. We train with
$$\min_\theta\ \mathbb{E}_{(x,y)}\Big[\,L_{\text{CE}}(x,y;\theta)+\lambda\max_{\delta\in\Delta}W_T\big(p_\theta(x+\delta),\,q_y\big)\Big],$$
where $\delta$ is a perturbation of the token embeddings inside a norm ball $\Delta$, $q_y$ is the gold label distribution, and $\lambda$ trades clean fit against robustness.
The new element is the inner maximization. The attacker climbs taxonomic cost rather than cross-entropy, so the training signal concentrates on the cross-branch flips that matter. The model is not pushed to be invariant across fine siblings, which is what the task needs it to tell apart.
Relation to Prior Work
Two papers motivate the idea. DAFA shows on image benchmarks that a class's adversarial vulnerability depends on how close its neighbors are. It reweights losses and widens perturbation budgets using closeness estimated from softmax confusion, raising worst-class robust accuracy on CIFAR-10 from 21.31% to 30.53%. That work is single-label and vision-only, and it ignores any external label structure.
Wasserstein Adversarial Regularization replaces the KL divergence in virtual adversarial training (a label-free method that penalizes prediction changes under the most damaging small perturbation) with a Wasserstein distance over a word2vec class-cost matrix. It uses this to resist training-label noise. It does not study test-time attacks, hierarchies, or text. We reuse its central move, a semantic cost inside the adversarial term, but we put that cost in the attacker's objective and target evasion robustness.
Two papers come closest within language tasks. Masked VAT for fine-grained entity typing restricts virtual adversarial smoothing to a flat candidate set from distant supervision, with the goal of noise robustness. It reports that unrestricted VAT hurts, which constrains our design: label-agnostic smoothing is not a viable baseline to beat cheaply. HiAdv uses "adversarial" in the GAN sense: a discriminator aligns text representations with a hierarchy-aware oracle, giving NYT Macro-F1 of 70.78 against 70.14 for its HPT backbone. It offers no defense against input edits. Neither paper weights errors by taxonomic distance, and neither evaluates against attacks.
TextFooler supplies the threat model. It shows that synonym substitutions reduce BERT's accuracy from 90.9% to 13.6% on IMDB, while human raters judged the original label preserved 92% of the time.
Supervised Tree-Wasserstein Distance supplies a method for learning tree weights by contrastive training, which we use as a variant when the given tree is untrustworthy. The label-clustering work on ultra-fine typing warns that embedding-derived structure mixes topical relatedness (grape–wine) with true type hierarchy (grape–fruit). We therefore default to curated hierarchies rather than embedding clusters.
Research Program
We first build a severity-aware evaluation. The standard robust accuracy is joined by expected taxonomic cost under attack, defined as the mean $W_T$ between attacked predictions and gold labels, and by a "severe-flip rate", the fraction of attacked examples whose prediction leaves the gold top-level branch.
Attacks come in two kinds. The first is the usual cross-entropy-driven TextFooler. The second is a cost-aware TextFooler whose word-importance ranking and substitution choice maximize $W_T$. Embedding-space projected gradient attacks (PGD, repeated small gradient steps projected back into the norm ball) use either objective. Evaluating cost-trained models against cost-aware attacks prevents an inflated result caused by a mismatched attacker.
We then train BERT-base and RoBERTa-large classifiers under each objective. The analysis asks where robustness gains land by depth and subtree, mirroring DAFA's per-class analysis. It also asks whether fine-sibling separability in representation space is preserved, which tests the smoothing-versus-discrimination tension. A secondary variant scales the per-example perturbation budget by how crowded the gold label's subtree is, a taxonomy analogue of DAFA's margin rule. This variant is reported only if it adds to the cost-weighted objective.
Baselines
The naive reference is standard fine-tuning with cross-entropy and no adversarial training, the HPT backbone for hierarchical classification. It sets clean accuracy and the floor for robustness. HiAdv reports 70.14 Macro-F1 on NYT for HPT and 70.78 for HiAdv. Neither paper reports results under attack, so for NYT, RCV1-V2, and WOS under TextFooler there is no reported result found on Lacuna. We will measure these ourselves.
The strongest competitor is standard embedding-space adversarial training. It uses the same PGD perturbation set and step count but a cross-entropy inner objective, on the same backbone. It is the strongest competitor because it is what practitioners use, and it may already remove most severe flips as a side effect. It receives the same data, epochs, number of attack steps, and the same hyperparameter search budget over $\lambda$ and the perturbation radius.
A second strong competitor is a WAR-style regularizer applied as a robustness defense. This is virtual adversarial training with the same $W_T$ cost but a label-free inner objective. It isolates whether using the gold label in the attacker's objective matters. A third is a DAFA port in which closeness between classes comes from running softmax confusion instead of the tree. This isolates whether the external taxonomy carries information beyond what the model's own confusions reveal.
The key ablation keeps the tree's shape but randomly permutes its labels. If the gains survive, they come from the cost's form rather than its semantics, and the claim fails.
Beating the strongest competitor means two things, judged at matched clean Macro-F1 (within 0.5 points) and under the cost-aware attack. Expected taxonomic cost and severe-flip rate must be clearly lower across seeds. Clean Macro-F1 on the deepest levels must be no worse.
Why not just…?
Why not just run standard adversarial training and report the severity metric? This is the most serious threat. Cross-entropy adversarial training might already eliminate cross-branch flips, leaving only harmless sibling flips. It is our main baseline. If its severity profile matches ours at equal clean accuracy, the proposal reduces to an evaluation contribution, and we would report it that way.
Why not just fix outputs afterward with hierarchy consistency rules or label clustering? PKL post-processing improves clean UFET F1 from 50.0 to 51.8 by removing conflicting neighbor labels. Post-processing, however, only filters the attacked logits. If an attack moves mass wholesale into the wrong branch, the filtered output is consistent but wrong. We will apply consistency rules on top of every model to measure this directly.
Why not just use the WAR regularizer, which already uses a semantic cost? WAR targets label noise, and its inner step ignores the gold label. An adversary at test time, by contrast, knows which direction is most damaging. The label-free variant is run as a baseline, so this question is answered empirically.
Why not just learn closeness from confusions, as DAFA does? Confusions are cheap and need no ontology, and they may be enough. The DAFA-port baseline and the permuted-tree ablation together tell us whether the curated taxonomy contributes anything.
Proposed Experiments
The main testbeds are hierarchical text classification on NYT (166 classes, depth 8), RCV1-V2 (103 classes, depth 4), and WOS (141 classes, depth 2). The deep NYT tree is where we expect the largest effect, and the shallow WOS tree is a control where the effect should shrink. A second testbed is entity typing on OntoNotes (89-class hierarchy) and BBN (47 classes). Masked VAT reports an average 3.8% Strict Accuracy gain there but no absolute values. UFET is a stretch setting, using its hierarchy-free label set with tree weights learned by supervised tree-Wasserstein.
We report clean Micro- and Macro-F1 by depth, robust accuracy, expected taxonomic cost, and severe-flip rate under all four attacks, with five seeds and training-time overhead.
The decisive checks are three. First, the cost-weighted objective must lower severe-flip rate relative to cross-entropy adversarial training under the cost-aware attack, at matched clean accuracy. Second, its deep-level clean Macro-F1 must degrade less. Third, both effects must vanish under the permuted tree. Additionally, the effect should grow with tree depth from WOS to NYT. A small human study, in the style of TextFooler's label-preservation check, will verify that low-cost flips are in fact judged acceptable, so the metric tracks real harm rather than tree geometry alone.
References
- DAFA: Distance-Aware Fair Adversarial Training
- Wasserstein Adversarial Regularization for Learning With Label Noise
- Alleviate Dataset Shift Problem in Fine-grained Entity Typing with Virtual Adversarial Training
- Utilizing Local Hierarchy with Adversarial Training for Hierarchical Text Classification
- Is BERT Really Robust? Natural Language Attack on Text Classification and Entailment
- Supervised Tree-Wasserstein Distance
- Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy