Is the Robustness Tax Paid in Features Humans Cannot See?

Three findings that are usually discussed separately may describe a single phenomenon. Past a certain point, image classifiers that get more accurate on ImageNet become worse at judging image similarity the way people do. They also become worse at predicting neural responses in the monkey brain's object-recognition area. Separately, training a classifier to resist adversarial attacks lowers its accuracy on clean images, yet the same kind of training makes it a better judge of perceptual similarity. Adversarial attacks here are tiny, deliberately chosen pixel changes that flip a model's output. This note proposes to test one hypothesis that would explain all of these results. The accuracy a classifier gains after its perceptual-alignment peak comes from a specific set of internal features that humans and primate cortex do not use. Adversarial training removes exactly these features, and their removal is the clean-accuracy loss people call the "robustness tax."

The test is direct. We record a classifier's internal features at its perceptual peak and again at the end of training, and isolate the directions it learned in between. We then remove only those directions. If the hypothesis is right, that single cut should do four things at once: restore human-like similarity judgments, restore prediction of primate neural responses, raise adversarial robustness, and lower clean accuracy by about the same amount that adversarial training does. A random cut of equal size should do none of these. If the hypothesis holds, the robustness–accuracy trade-off can be described more precisely: it costs the accuracy that comes from features humans cannot perceive. If it fails, we learn that these failures have separate causes, which is also worth knowing.

Core Claim

The motivating evidence comes from three places. First, Do better ImageNet classifiers assess perceptual similarity better? swept 722 classifiers and found an inverse-U relationship. Perceptual alignment on the Berkeley-Adobe patch dataset (BAPPS), measured as human two-alternative forced-choice agreement, peaks at 20–40% ImageNet accuracy. Past the peak, it is anti-correlated with accuracy at $r=-0.84$. The authors ruled out spatial-frequency sensitivity and distortion invariance as explanations and left the mechanism open. Second, Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex found the same decoupling for macaque inferotemporal (IT) cortex predictivity, with the break near 80% top-1 accuracy, and attributes it to non-biological shortcut features. Third, on the robustness side, Learning Perceptually-Aligned Representations via Adversarial Robustness and Model metamers reveal divergent invariances between biological and artificial neural networks show that adversarial training makes a model's representations human-interpretable. Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics shows that robust CLIP models are better zero-shot perceptual metrics. Yet Theoretically Principled Trade-off between Robustness and Accuracy (TRADES) formalizes why robustness costs clean accuracy.

No paper in this set asks whether the post-peak accuracy in the first two findings and the accuracy lost in the third are the same accuracy. Our claim is that they are, and that this is a quantitative, falsifiable identity rather than a loose analogy. Applied to a standard model, the "late subspace" ablation defined below should move that model onto the same curve that adversarially trained models occupy. That curve relates clean accuracy, perceptual score, IT predictivity and robust accuracy.

Defining the late subspace

Fix a standard training run. Let $h_t(x)$ be the features at a chosen layer for image $x$ at checkpoint $t$. Let $t^*$ be the checkpoint where the BAPPS perceptual score peaks, measured on a held-out split, and let $T$ be the final checkpoint. We fit a linear map $A$ that predicts final features from peak features over training images. We then take the residual

$$r(x) = h_T(x) - A\,h_{t^*}(x),$$

which is the part of the final representation that is not a linear re-encoding of what the network already knew at its perceptual peak. The top $k$ principal directions of $r$ form a matrix $U_k$. The ablated model replaces $h_T$ with $(I - U_kU_k^\top)h_T$ and retrains only the linear classifier head.

The linear map matters because networks rotate and rescale their features during training. Removing everything that changed would conflate re-encoding with genuinely new features. We sweep $k$ and repeat the procedure at several depths, because the perceptual metric draws on early and middle layers while classification draws on late ones.

Research Program

The first stage builds a matched model grid. We train ResNet-18 on Tiny ImageNet densely across TRADES strengths $\lambda$ and across training length, saving checkpoints throughout. We train a smaller set of ResNet-50 models on ImageNet at three or four adversarial budgets, following the setup of Learning Perceptually-Aligned Representations via Adversarial Robustness. Every checkpoint is scored on five axes:

  • clean accuracy;
  • robust accuracy under PGD and AutoAttack;
  • perceptual score on BAPPS using the Kettunen et al. protocol, with 2AFC agreement on NIGHTS as a second perceptual test;
  • Brain-Score IT and V1 predictivity;
  • metamer recognizability, measured with the synthesis procedure of Feather et al. and a small human recognition study.

From this grid we can see whether the inverse-U flattens or shifts as $\lambda$ grows. That is the first prediction: robust models should keep gaining perceptual score as accuracy rises, because they do not acquire the late, non-perceptual features.

The second stage is the accounting test. At matched training compute, we compare two quantities: the clean accuracy that TRADES at strength $\lambda$ gives up, and the accuracy a standard run gains between $t^*$ and $T$. The hypothesis predicts that robust models plateau near the accuracy of the standard model at its perceptual peak. If adversarial training costs far less or far more than the post-peak gain, the identity is wrong.

The third stage is the causal ablation described above. The hypothesis predicts that it moves a standard model toward the robust models on all five axes at once. We also turn the result into something usable. If a single backbone can serve two roles, the projected features become a perceptual metric with a learned linear head, and the full features remain the classifier. We then attack that metric with the rank-flip suite of Attacking Perceptual Similarity Metrics, which uses FGSM, PGD and spatial-warp attacks, to see whether removing the late subspace also removes the attack surface.

Two prior results constrain the design, and we treat them as risks. First, Feather et al. report that metamer recognizability dissociates from both Brain-Score and adversarial vulnerability. Some moderately robust models still produce unrecognizable metamers. So we predict metamer recovery only weakly, and a failure on that axis alone would limit the claim rather than refute it. Second, Do Perceptually Aligned Gradients Imply Robustness? obtains robustness by forcing input gradients toward diffusion-model score directions. This suggests the relevant property may live in how the network responds to small input changes rather than in which feature directions it uses. To check this, we also measure input-gradient alignment before and after ablation. If ablation improves perceptual score without changing gradients or robustness, the "same features" claim fails, and the gradient account is favored instead.

Baselines

The naive reference is the unmodified final standard checkpoint, evaluated on all five axes. Every claimed movement is measured relative to it.

The early-stopped peak checkpoint is the second reference. Kettunen et al. report a BAPPS perceptual score of 70.2 for an early-stopped ResNet-200 and 69.1 for ResNet-6, against 68.9 for AlexNet and 67.0 for VGG. Our ablated model counts as better than early stopping only if two things hold together. It must match or exceed the early-stopped checkpoint's perceptual score. And the full, unprojected backbone, which carries the same weights, must retain the final checkpoint's classification accuracy.

TRADES is the strongest competitor for the robustness question. It is the method that defines the trade-off, and it gives a tunable family of models. On CIFAR-10 it reports a smooth transition from 88.64% natural accuracy to 56.61% PGD robust accuracy (Zhang et al.). We run TRADES on the same architectures, data and number of epochs as the standard runs. We tune $\lambda$ over the same grid size used to tune $k$, and we report Pareto curves rather than single points. On the ImageNet-scale BAPPS perceptual score of TRADES-trained ResNets, there is no reported result found on Lacuna, so we measure it ourselves. The ablation does not need to beat TRADES on robust accuracy. The hypothesis is confirmed if the ablated standard model lands on the TRADES curve in the joint space of clean accuracy, perceptual score and IT predictivity.

Robust CLIP is the strongest existing metric. $R\text{-CLIP}_F$ reaches 90.6% clean and 74.3% robust 2AFC accuracy on NIGHTS (Croce et al.). LipSim reaches 85.6% clean and holds 72.20% under APGD at $\epsilon=2.0$, while DreamSim, at 96.2% clean, falls below 1% (LipSim). We do not expect a ResNet-50 to beat CLIP-scale models in absolute terms. We report these numbers as context and compare at matched backbone size. We also evaluate $R\text{-CLIP}_F$ on Brain-Score, which its authors did not do. For neural predictivity, the reference values are $r^2=0.41$ for standard MaxViT and $0.54$ for the human-saliency-harmonized MaxViT (Linsley et al.).

Three ablations isolate the key ingredient. The first removes a random subspace of the same dimension $k$. The second removes the top-variance directions of $h_T$ itself, with no temporal reference. The third removes a high-spatial-frequency subspace. Kettunen et al. already argue frequency is not the explanation, so we expect this one to fail. The proposal's specific ingredient is temporal localization, meaning the features learned after the perceptual peak. If any of the three controls matches the late-subspace ablation, that ingredient is not doing the work.

Why not just…?

Why not just adversarially train and accept the cost? Adversarial training is not a rival to this proposal; it is the reference the hypothesis is tested against. Its known drawbacks are that it is often about 10× slower than standard training (Zhang et al.) and that it overfits to its training budget. Ganz et al. report that an $L_2$ adversarially trained model falls to 18.81% at $\epsilon=1.0$, while their gradient-aligned model holds 30.43%. The real threat is that the ablation turns out to be a worse version of TRADES. In that case, the practical recommendation remains "adversarially train." The accounting test still decides whether the tax is paid in perceptually invisible features, and that question is what this work is designed to answer.

Why not just early-stop or use a shallow network as the metric? Early stopping does give a good metric, at 70.2 perceptual score per Kettunen et al. But it leaves a separate, poor classifier. Shallow backbones like AlexNet are also exactly what Ding et al. show to be vulnerable to transferable rank-flip attacks. If our projected metric only matches the early checkpoint on clean perceptual score, while also matching it under attack, we will say so plainly: the deliverable then reduces to the scientific identification of the late subspace, not a better metric.

Why not just fine-tune a robust foundation model? Croce et al. show this is the best route to a robust metric. It explains nothing about the trade-off, however, and the underlying classifier still pays the usual accuracy cost. Linsley et al. also show that scaling deepens the divergence from IT cortex. Our evaluation of $R\text{-CLIP}_F$ on Brain-Score checks whether robustness at scale also recovers neural alignment. If it does, that is independent support for the hypothesis.

Why not just filter high frequencies or add biological front-ends? Kettunen et al. rule out frequency sensitivity as the cause of the inverse-U, so we include frequency removal only as a control. Biological front-ends such as the divisive normalization in Cirincione et al. improve robustness to weather and contrast corruptions on Tiny ImageNet-C while keeping 99% of relative clean accuracy. They are untested against worst-case attacks, though, and they act only at the input stage. The hypothesis locates the problem in late-acquired features, so a front-end change should not touch it. We test this by running the full grid with a divisive-normalization front-end as well.

Decisive Checks

The hypothesis lives or dies on three results. First, in the Tiny ImageNet and ImageNet grids, the perceptual-score peak should move to higher accuracy as $\lambda$ grows. Second, the late-subspace ablation of a standard ResNet-50 should raise its BAPPS perceptual score above the final checkpoint and toward the peak. It should raise IT predictivity, raise PGD robust accuracy well above zero, and lower clean accuracy to within a few points of the peak-checkpoint accuracy. The random-subspace, top-variance and frequency controls should each fail on at least the perceptual and robustness axes. Third, the projected metric, attacked end to end with the methods of Ding et al., should show fewer rank flips on BAPPS and NIGHTS than LPIPS-style metrics on the same backbone.

The hypothesis is refuted if the ablation restores perceptual score but leaves robustness unchanged, or if the controls perform as well. In that case, perceptual misalignment and the robustness tax have separate causes despite their coincident timing, and the field should stop treating adversarial training as a route to human-like vision.