Splitting the Robustness Tax of Causal Defenses

Training a neural network to resist small adversarial input changes usually costs accuracy on ordinary, unperturbed images. Causal defenses are methods that try to make a network rely only on features that actually cause the label and ignore coincidental correlations. They were expected to escape this cost, since causal features should be both predictive and stable. In vision they have not. Adversarial Visual Robustness by Causal Intervention (CiiV) drops CIFAR-10 clean accuracy from about 94% to about 87%. Causal Information Bottleneck Boosts Adversarial Robustness of Deep Neural Network (CausalIB) reports a clean drop on CIFAR-100 that its authors call unresolved. On graphs, by contrast, IDEA: Invariant Causal Defense for Graph Adversarial Robustness pays essentially nothing: it reaches 88.4% clean accuracy on Cora, matching undefended networks.

This note argues that "clean accuracy loss" in these papers mixes two different costs that need to be measured separately. One is a shortcut tax: the defense stops using a spurious cue, and a test set that shares the cue punishes it for that. This is the cost the defense is supposed to pay. The other is a content tax: the defense also suppresses or distorts features that genuinely carry the label. That cost is pure loss. We propose a simple diagnostic that separates the two, a controlled experiment that predicts when each appears, and a targeted training method. The method applies the causal invariance penalty only to features flagged as shortcut-driven and uses accuracy-aware adversarial training everywhere else. The aim is to explain why the tax vanishes on graphs but not on images, and to test whether removing the content tax lets a causal defense match the best geometric adversarial training on clean data while beating it under distribution shift.

Core Claim

The claim has a measurable form. Split any clean test set into a shortcut-aligned group $A$ and a shortcut-conflicting group $C$. In group $A$ a spurious attribute agrees with the label. In group $C$ it disagrees. Compare a defended model to its undefended counterpart on each group.

If the defense only removes shortcuts, accuracy should fall on $A$ but rise on $C$, because the model no longer gets misled there. If the defense also damages causal content, accuracy falls on $C$ as well. The sign and size of the change on $C$ is therefore a direct estimate of the content tax, and the remaining change on $A$ is the shortcut tax.

Our hypothesis comes from the robust error bound in Theoretically Principled Trade-off between Robustness and Accuracy (TRADES). That paper shows robust error is at most clean error plus a "boundary error" term, the share of points lying within the perturbation radius of the decision boundary. A forced trade-off arises only when the small, perturbation-sensitive signal that drives clean accuracy is itself label-relevant. Two cases follow:

  • Sensitive signal lives in spurious factors. Causal invariance and robustness push in the same direction, and only a shortcut tax appears.
  • Sensitive signal lives in the causal content. Indiscriminate invariance penalties of the kind CiiV and CausalIB use must incur a content tax.

IDEA's graph setting is a natural instance of the first case, because adversarial edges are non-causal by construction. CIFAR-style images, which lack strong annotated shortcuts, should look like the second case. The method follows from this. If we can tell which internal features are shortcut-driven, we can apply invariance there and leave content features to a defense designed to keep the decision boundary intact.

Research Program

Part 1: a controlled two-by-two study on synthetic images with a known causal structure. We extend Colored-MNIST (digit images whose color is spuriously correlated with the label) along two axes:

  • whether the shape is causal and the color spurious, or the reverse;
  • whether a small high-frequency signal readable within an $\epsilon$-ball is attached to the causal factor or to the spurious one.

Because both the causal graph and the location of the perturbation-sensitive signal are known, we can compute the aligned/conflicting decomposition exactly. We then test whether the content tax appears only when the sensitive signal is causal.

Part 2: the same decomposition applied to existing defenses on real benchmarks. We use CelebA, where an annotated attribute can serve as the spurious cue. GLEAN used CelebA but the file records no number for it, so this measurement is new. We also use DomainNet, where domain identity plays the spurious role. CIFAR-10 and CIFAR-100 serve as a control with no annotated shortcut. There we approximate group $C$ with images where an early-stopped, shortcut-prone model is confidently wrong, and we report this proxy separately because it is weaker.

Part 3: the targeted method. It reuses the dual-model idea from Backdoor Defense via Deconfounded Representation Learning (CBD). That paper trains a deliberately early-stopped model that soaks up fast-learned confounding features, then trains a clean model to share little information with it. CBD achieves a 1.6% average attack success rate with clean accuracy close to a model trained on fully clean data, though only for training-set poisoning.

We use the early-stopped model differently: as a detector. Each channel of the main network gets a nuisance score, which is how much of its activation the early-stopped model's features explain. On high-scoring channels we apply CausalIB's penalty, which penalizes the variance of a channel's direct effect on the prediction under injected noise. All other channels are trained with the accuracy-aware inner maximization of A3T: accuracy aware adversarial training. For misclassified points, that procedure attacks the predicted label rather than the true one, so wrong points are not pushed deeper into wrong classes.

The single new ingredient is the routing: invariance where the causal score says "shortcut", boundary-preserving robustness elsewhere. CBD's own limitation constrains this design. If shortcut features are not learned faster than content features, the detector fails. Part 1 deliberately includes a configuration where the spurious signal learns as slowly as the causal one, so we can see the method break in a controlled way rather than discover it later.

Baselines

The naive reference point is ordinary training with no defense, run on the same backbone and augmentations. It anchors both the aligned/conflicting decomposition and the size of every tax.

Standard projected gradient descent (PGD) adversarial training and TRADES are the canonical trade-off methods. TRADES reports 88.64% natural accuracy with tuned $\lambda$ and 56.61% white-box PGD robust accuracy on CIFAR-10. We sweep TRADES's $\lambda$ over a fixed grid so it is compared as a full curve of operating points, not a single setting.

The strongest existing competitor is A3T. On CIFAR-10 it reports 88.98% natural accuracy, against 87.30% for standard adversarial training, and its A3T+ variant reports 55.59% AutoAttack accuracy. These are the clean and robust numbers our method must reach. We also include TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training, which restricts perturbations to directions along the data manifold. On WideResNet it reports clean accuracy improving from 81.76% to 83.73% with AutoAttack accuracy at 36.58%. TART is the closest geometric analogue to our routing, with "on the manifold" playing the role that "causal" plays for us.

The causal baselines are:

  • CiiV: 43.23% AutoAttack accuracy without adversarial training, at roughly 87% clean accuracy.
  • CausalIB: 35.89% PGD-20 accuracy on CIFAR-10.
  • CBD: its deconfounding objective applied directly to evasion.
  • Certified Causal Defense with Generalizable Robustness (GLEAN): used on Colored-MNIST and DomainNet, where it reports an average certified radius of 0.2466 versus 0.0458 for Gaussian smoothing, and 63.0% versus 33.0% certified accuracy at radius zero on DomainNet. It requires environment labels, which we provide only to GLEAN and record as an advantage.

All methods are retrained by us on one backbone per benchmark, with the same epochs and the same attack budget. Each gets the same number of hyperparameter trials, selected on a validation split with the same robust-plus-clean criterion. Published numbers are cited only as sanity checks, since architectures differ across papers.

Two ablations isolate the routing. In the first, the same fraction of channels is penalized but chosen at random. In the second, all channels are penalized, which is CausalIB-style penalization layered on A3T.

Beating A3T means two things. First, on CIFAR-10 and CIFAR-100, matching its clean and AutoAttack accuracy within seed noise. Second, on CelebA, DomainNet and Colored-MNIST, exceeding it on shortcut-conflicting clean accuracy and on robust accuracy under shift at equal in-distribution clean accuracy.

Why not just…?

Why not just use A3T or TART, which already reduce clean degradation without any causal machinery? This is the real threat. On CIFAR-10, A3T outperforms every causal defense in the file on both clean and robust accuracy. We therefore do not claim to beat it there. The claim is that geometric methods control where the boundary moves in input space, but not which features the model relies on. They should show no gain on shortcut-conflicting examples, and the collapse of non-causal smoothing under shift reported by GLEAN suggests they degrade under shift. If A3T matches our method on group $C$ of CelebA and DomainNet, the causal component adds nothing and we will report that.

Why not just tune the regularization weight of CausalIB, CiiV or TRADES to pick a better operating point? Tuning one weight moves a model along its existing trade-off curve. It cannot change which features absorb the penalty. Our decomposition makes this testable. If the uniformly penalized ablation, swept over its weight, ever reaches the same group-$C$ accuracy and robust accuracy as the routed method, the gain came from penalty strength rather than targeting.

Why not just reweight or augment against the known shortcut? Upweighting conflicting examples or randomizing color addresses annotated shortcuts, but it needs those annotations. Color Backdoor: A Robust Poisoning Attack in Color Space reports that aggressive color shifting itself reduces clean accuracy. CiiV reports that Mixup-trained models drop to 0% under AutoAttack. We still run oracle group reweighting on top of adversarial training as an upper reference on the annotated benchmarks. If it dominates, the honest conclusion is that the detector is only useful where annotations are absent.

Decisive Experiments

Decomposition check. On Colored-MNIST, CelebA and DomainNet, report each defense's accuracy change on groups $A$ and $C$, together with AutoAttack and PGD robust accuracy, over three seeds. The core claim fails if CiiV's and CausalIB's group-$C$ change is zero or positive on CIFAR-style data. That would mean their tax is not a content tax at all.

Controlled prediction. In the synthetic two-by-two study, a content tax should appear only in the cells where the perturbation-sensitive signal is causal. Seeing a tax in the spurious-signal cells would contradict the reading of the TRADES bound used here.

Method check. - On CIFAR-10 and CIFAR-100, the routed method must stay within seed noise of A3T on clean and AutoAttack accuracy. - On the shift benchmarks, it must beat A3T and the random-routing ablation on group-$C$ accuracy and robust accuracy under shift. - The slow-shortcut synthetic configuration should show the predicted failure of the early-stopped detector.

Cost. We report wall-clock overhead relative to A3T. CBD reports 10–20% overhead for its dual model, which sets the expected cost.