Making a pre-trained vision-language model such as CLIP (an image encoder and a text encoder trained to agree on matching image-caption pairs) robust to adversarial inputs means fine-tuning it on attacked images. Attacked images are inputs nudged by a small, deliberately chosen perturbation that flips the prediction. This fine-tuning has two known costs. First, it destroys the model's ability to recognize categories it was never fine-tuned on, a form of catastrophic forgetting. Second, building each attacked image takes many gradient steps, so training runs ten to twenty times slower than ordinary fine-tuning. The strongest current fix for forgetting, PMG-AFT, keeps a frozen copy of the original model as a teacher and pulls the fine-tuned model's full class distribution toward it. It still pays the full multi-step attack cost, plus an extra teacher pass reported at about 300 seconds per epoch.
We propose to ask the frozen model for something more structured than its raw predictions. It can supply a class taxonomy: groups of semantically related classes, obtained by clustering its own text embeddings of class names. We then anchor only the coarse group-level distribution to the teacher. The fine-tuned model may reshuffle probability among sibling classes, such as two dog breeds, but may not move mass from "animal" to "vehicle." Our bet is that this one structural change does two jobs. It prevents forgetting with less over-constraint than a flat anchor. It also stabilizes cheap single-step attacks, which otherwise suffer a sudden collapse called catastrophic overfitting. No human-curated tree is needed at any point.
Core Claim
Hierarchy-aware robustness methods depend on a human-made tree. CHAT trains coarse-to-fine along an 8-level biological tree on iNaturalist-H. HAFrame fixes classifier geometry to lowest-common-ancestor distances in a given taxonomy. Both are unusable where such trees do not exist, which is the "domain availability gap" that Deng et al. document. In robust adaptation of a foundation model, however, a usable taxonomy is already sitting in the frozen teacher, and it does not drift, because the teacher is never updated.
Our claim has two parts. First, a coarse-level distillation anchor built from this induced taxonomy retains zero-shot accuracy at least as well as PMG-AFT's flat anchor. Second, combined with sibling-aware soft labels, it lets single-step adversarial fine-tuning reach robustness comparable to multi-step PMG-AFT at a fraction of the wall-clock cost, without catastrophic overfitting.
The closest work is PMG-AFT, which is flat, multi-step and expensive. The second closest is TDAT, which stabilizes single-step training from scratch using a taxonomy of sample behaviors (whether the clean and attacked predictions are right or wrong). TDAT uses no semantic class structure and no pre-trained anchor. Our proposal differs from both in where the structure comes from and in what it regularizes.
Research Program
The taxonomy is built once, before training. We embed each class name of the fine-tuning set with the frozen text encoder, using standard prompt templates. Optionally, we mix in the teacher's zero-shot confusion matrix on clean training images. We then run agglomerative clustering and cut the resulting tree at a small number of levels. Let $G$ be the set of groups at one level. For any class-probability vector $p$, define the group distribution $p^G_g = \sum_{c \in g} p_c$.
Training uses single-step attacks: one signed-gradient step from a random start, the FGSM recipe. Let $x$ be a clean image, $\delta$ its single-step perturbation, $p_S$ the fine-tuned model's prediction, $p_T$ the frozen teacher's prediction, and $\tilde y$ a soft label. The objective is
$$\mathcal{L} = \mathrm{CE}\big(p_S(x+\delta), \tilde y\big) + \lambda \sum_{\ell} \mathrm{KL}\big(p_T^{G_\ell}(x) \,\|\, p_S^{G_\ell}(x+\delta)\big).$$
Here $\ell$ ranges over the tree levels and $\lambda$ weights the anchor. The soft label $\tilde y$ puts most mass on the true class and spreads a small amount over its siblings in the induced tree, rather than over all classes as ordinary label smoothing does. This is the semantic counterpart of TDAT's dynamic label relaxation. The rationale is that the label-flipping events TDAT links to catastrophic overfitting should be less violent when the target already tolerates sibling confusion.
There is no fine-level KL term. The question is whether giving up fine-level anchoring costs retention, or instead frees the model to fit the robust task.
Three analyses accompany the method. First, we track multi-step robust accuracy every epoch, to detect catastrophic overfitting, the sudden collapse of robustness that TDAT reports around epoch 20 for plain FGSM training on CIFAR-10. Second, on datasets that do have human trees, we measure how well the induced tree agrees with them, to explain when induction suffices. Third, we measure mistake severity under attack, meaning the tree distance between the predicted and true class, scored with human trees used only for evaluation. This asks whether coarse anchoring keeps attacked errors semantically local, the property CHAT and HAFrame obtain with curated trees.
Baselines
The naive reference is the original frozen CLIP evaluated zero-shot. Alongside it we run standard adversarial fine-tuning with multi-step attacks and no anchor. PMG-AFT reports that this recipe collapses clean and zero-shot accuracy. The frozen model shows the ceiling for retention and the floor for robustness. The unanchored fine-tune shows the forgetting we are trying to prevent.
The strongest existing competitor is PMG-AFT itself. It runs multi-step attacks during fine-tuning and adds a flat KL term to the frozen teacher plus a clean-feature alignment term. It is the strongest competitor because it addresses exactly our two targets, robustness and zero-shot retention, on the same evaluation protocol. Its reported figures are 17.91% average robust accuracy under AutoAttack versus 1.74% for the original CLIP, an average robust gain of 4.99% over prior methods, and an average clean gain of 8.72%. These come from fine-tuning on TinyImageNet and testing on 15 zero-shot datasets.
We will rerun PMG-AFT with its published recipe on the same backbone, fine-tuning data, perturbation budget and number of epochs as our method. Both methods get the same hyperparameter search budget. Beating it means one of two outcomes. Either we match its average robust and clean accuracy within about one point while using substantially less wall-clock time, with a target of at least 3× less. Or we exceed it at equal wall-clock time. We report GPU-hours alongside accuracy for every run.
Three ablations isolate the key ingredient. The first is PMG-AFT's flat anchor trained with single-step attacks. It tests whether any teacher anchor prevents catastrophic overfitting, and it is the most dangerous comparison. The second is our coarse anchor with a random grouping of the same group sizes. It tests whether the semantic structure, rather than the act of aggregating probabilities, does the work. The third is our method with ordinary uniform label smoothing in place of sibling-aware smoothing.
Where a human tree exists, we add a fourth variant that substitutes it for the induced tree. This tells us how much is lost by not curating.
We also report TDAT as the strongest fast-training method that uses no anchor. It is run single-step on the same CLIP fine-tuning setup, to show whether behavioral stabilization alone already preserves zero-shot transfer. For TDAT, the only reported figure in the file is up to a 1.62% gain in robust accuracy under 10-step attacks on CIFAR-100, from from-scratch training. No reported result was found on Lacuna for TDAT applied to CLIP fine-tuning.
Why not just…?
Why not just run PMG-AFT with single-step attacks? This is the real threat. A flat teacher anchor might by itself stop the loss landscape from distorting in the way that causes catastrophic overfitting. If so, the coarse anchor adds nothing on the speed axis. We have no evidence in hand that settles this, so we treat it as the first experiment.
If flat-anchor single-step training stays stable and matches our robustness and retention, the paper's contribution narrows to a single question: does the coarse anchor relax over-constraint on the fine-tuning task, measured as robust accuracy on TinyImageNet itself, while keeping equal zero-shot retention? We would report that honestly as the result.
Why not just use a human taxonomy such as WordNet, as CHAT and HAFrame do? Where one exists, we run that variant. Human trees, however, are absent for many of the 15 zero-shot transfer targets and for most specialist domains. Text2Tree notes that hierarchy methods fail on flat label spaces without heavy manual engineering. If the induced tree performs within noise of the human tree on CIFAR-100 and tiered-ImageNet-H, curation buys nothing for this purpose. If it falls clearly short, we learn which kinds of semantic structure the text encoder misses.
Why not just freeze the backbone and train adapters or LoRA adversarially? Low-rank adaptation trains small add-on weight matrices while the original weights stay frozen. This limits forgetting, but it does not remove the attack cost. Generating each perturbation still needs gradients with respect to the input, backpropagated through the whole network at every attack step. Weight freezing is orthogonal to our method and could be combined with it. It answers the forgetting question but not the overhead question.
Why not just use robustness tokens or a robustness prompt, which cost almost nothing? Robustness Tokens train about 10 extra input tokens appended to a frozen vision transformer. Bhagwatkar et al. instead add a text prompt warning the model that the image may be perturbed. Both are cheap, but both authors concede that an attacker who knows or optimizes against the tokens or prompt can bypass them.
We evaluate under full white-box attacks, where the attacker has the complete fine-tuned model. We will include the token defense under an adaptive attack that also has access to the tokens, to make this contrast concrete.
Why not just put a robust purifier in front of the frozen model? CRoPD trains a robust auto-encoder that cleans images before they reach the untouched foundation model. It cuts training on ImagenetTe from over 22 hours to 1.7 hours. Its authors note that it does not match full adversarial fine-tuning and that it weakens in data-scarce settings. We include it as the cheapest-compute reference point.
Proposed Experiments
The main experiment follows PMG-AFT's protocol. We adversarially fine-tune CLIP on TinyImageNet, then evaluate clean and robust zero-shot accuracy across the 15 transfer datasets, including CIFAR-10, CIFAR-100, EuroSAT and FGVC-Aircraft. Robustness is measured under 10-step attacks and AutoAttack. Every result is reported against GPU-hours.
The first decisive check is the catastrophic-overfitting curve. We plot per-epoch multi-step robust accuracy for four runs: plain single-step training, single-step training with the flat anchor, single-step training with the random-group anchor, and our method. If only the semantically grouped variants avoid collapse across several perturbation budgets and seeds, the structural claim holds. If the flat anchor also avoids it, we fall back to the narrower claim described above.
The second decisive check is mistake severity. Using the CIFAR-100 superclasses, tiered-ImageNet-H and the iNaturalist-H tree purely as scoring keys, we compare the average tree distance of errors under attack across methods. For context, HAFrame reports mistake severity of 6.89 versus 6.94 on tiered-ImageNet-H with a curated tree, and CHAT reports a 0.17 reduction in average mistake on iNaturalist-H. Our runs are not directly comparable to these, because those models were trained on the target classes rather than evaluated zero-shot. We will therefore compare against PMG-AFT under identical conditions rather than against those numbers.
The third check measures how agreement between the induced and human trees varies with tree level. We correlate it with the gap between the induced-tree and human-tree variants. This tests whether the frozen model's taxonomy is good enough exactly where it matters for retention.
The proposal fails cleanly if either of two things happens. Coarse anchoring may retain less zero-shot accuracy than flat anchoring at equal robustness. Or single-step training with our objective may still collapse at the perturbation budgets PMG-AFT uses.
References
- Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness
- A Hierarchical Assessment of Adversarial Severity
- Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severity
- Hierarchical semantic indexing for large scale image retrieval
- Taxonomy Driven Fast Adversarial Training
- Text2Tree: Aligning Text Representation to the Label Tree Hierarchy for Imbalanced Medical Classification
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design
- How to Enhance Downstream Adversarial Robustness (almost) without Touching the Pre-Trained Foundation Model?