How Much Adversarial Training Does Cross-Seed Explanation Agreement Need?

Train the same neural network twice with different random seeds and ask each copy which input features mattered most for a given prediction. The two copies often disagree. On a standard sentiment benchmark, text classifiers that differ only in their random seed share only about 53–56% of their top Integrated Gradients features (On the Lack of Robust Interpretability of Neural Text Classifiers). On small tabular datasets, this seed-driven disagreement is the main source of instability in SHAP explanations, larger than the random sampling noise inside the explainer itself (Explanation Multiplicity in SHAP: Characterization and Assessment). Adversarial training, which teaches a model to resist worst-case input perturbations, is known to make explanations sparser and more stable against input noise. It is also notoriously expensive, typically costing 10–40× standard training because every update needs an inner search for the worst perturbation.

This proposal asks a narrower and cheaper question: how much of that adversarial computation is needed to make independently trained models agree on their explanations? Our hypothesis is that agreement across seeds saturates at a small fraction of the compute that robust accuracy needs. One gradient step and a small perturbation radius may buy most of the agreement that full multi-step training buys. If that holds, practitioners can get reproducible explanations without paying for robustness they do not need. If it fails, the experiments will show whether plain noise augmentation, an explicit regularizer, or simply averaging several standard models is the better use of the same compute.

Core Claim

Seeds tend to disagree on features that carry weak, interchangeable predictive signal. Each run latches onto a different subset of these features. Adversarial training suppresses exactly such weak, non-robust features: for linear models it provably acts as a weight regularizer that drives weak features to zero (Concise Explanations of Neural Networks using Adversarial Training). We therefore predict two things. First, adversarial training reduces model-induced explanation multiplicity, which no paper in our file has measured. Second, this reduction is a low-dose effect. Removing the weakest features should require only a small perturbation radius $\epsilon$ and few inner steps. Robust accuracy against strong attacks is different: it needs the full budget and is vulnerable to catastrophic overfitting, where single-step training abruptly loses robustness to multi-step attacks.

We measure agreement with the dual-seed protocol of Hwang et al. Train-seed pairs and explainer-seed pairs are varied separately, so that model-induced and explainer-induced variance are kept apart. For two models $f_{s}$ and $f_{s'}$ trained with seeds $s, s'$, and an explainer $E$ run with a fixed explainer seed, cross-seed agreement at input $x$ is

$$ A(x) = \mathbb{E}_{s \neq s'}\left[\mathrm{RBO}\big(E(f_s, x),\, E(f_{s'}, x)\big)\right]. $$

Here RBO is Rank-Biased Overlap, a top-weighted similarity between two ranked lists. We also report top-$k$ Jaccard, because Hwang et al. show that $\ell_2$ distances between attribution vectors hide large reordering among the top features. The object of study is the curve of $A$, averaged over test inputs, against training compute. Compute is counted in forward-plus-backward-pass equivalents.

Relation to Prior Work

Three papers motivate the question but leave it open. Concise Explanations of Neural Networks using Adversarial Training shows that full PGD adversarial training (Projected Gradient Descent, the standard multi-step inner attack) raises attribution sparsity. On MNIST it raises the Gini index by 0.06 at under 1% accuracy cost. The paper names compute cost as a limitation and never varies seeds. Training for Stable Explanation for Free replaces adversarial training with R2ET, a regularizer costing two extra backward passes. R2ET widens the margin between top-$k$ and other attributions and penalizes the Hessian, the second derivative that measures how sharply gradients change. It targets attacks on the input, not retraining randomness, and its authors note that black-box explainers such as KernelSHAP are untested. Training for Trustworthy Saliency Maps constrains our design. It finds that adversarial training improves input-side stability but worsens output-side stability: explanations shift when logits move slightly, even if the label does not change. We therefore report its Relative Output Stability metric alongside agreement, so that we do not trade one instability for another.

The closest existing work is R2ET, and our proposal differs from it in two ways. It measures stability across retrained models rather than against input attacks, and it maps a compute–agreement curve rather than proposing a single new method. On the explainer side, Stabilizing Estimates of Shapley Values with Control Variates reduces KernelSHAP sampling noise at test time. On German Credit it cuts average rank changes from about 60 to about 20. It cannot touch seed variance, which Hwang et al. find dominant on that same dataset. We use ControlSHAP to remove explainer noise, so that the remaining disagreement can be attributed to the model.

Robustness Tokens shows that robustness can be bought cheaply by tuning ten prefix tokens on a frozen Vision Transformer. Because the backbone is frozen, seed variance there lives only in the tokens. That is a different and smaller question, which we leave to follow-up work.

Research Program

We train families of models at controlled compute levels with ten seeds per configuration. The configurations are:

  • standard training;
  • isotropic Gaussian input-noise augmentation, sweeping its noise scale;
  • single-step adversarial training with random initialization, sweeping $\epsilon$;
  • PGD with 2, 5, and 10 steps, sweeping $\epsilon$;
  • R2ET.

For text we perturb in embedding space, because discrete adversarial search is too costly inside a training loop (Towards Improving Adversarial Training of NLP Models).

Each model is explained with four methods:

  • KernelSHAP, both with and without ControlSHAP;
  • LIME;
  • Integrated Gradients;
  • SmoothGrad.

Explainer seeds are held fixed for the cross-seed comparison and varied for the explainer-noise comparison.

The mechanistic test runs on tabular data. For each feature we compute two quantities: its cross-seed attribution variance under standard training, and the drop in its attribution mass under low-$\epsilon$ adversarial training. The hypothesis predicts a strong positive rank correlation, meaning AT removes precisely the features that seeds disagree on. If the correlation is near zero, any agreement gain comes from something other than feature suppression, and the explanation in our Core Claim is wrong.

Baselines

The naive reference is standard training with the explainer run at its default budget. Its agreement level is the floor every method must beat. Slack et al. report Integrated Gradients top-attribution overlap of roughly 53–56% across seeds on SST-2, and we must first reproduce that number. On German Credit and ACS Income, the dual-seed paper reports qualitative dominance findings but no agreement value we can cite. We therefore establish these floors ourselves.

The strongest competitor is a compute-matched seed ensemble. We train $K$ standard models so that their total compute equals one adversarially trained model, then average their attributions. Averaging directly cancels seed noise, and it needs no adversary. Agreement is then measured between two independent ensembles. Ensembles produce a different kind of artifact from a single model, so we also report test-time cost: explaining $K$ models costs $K$ times more per query. We found no reported result on Lacuna for cross-seed agreement of seed ensembles.

R2ET is the strongest existing cheap training method. We run its released objective with the same tuning budget as our adversarial variants: a fixed grid over its two coefficients, chosen on validation agreement. On the Bank dataset it reports nearly 100% Precision@k under the ERAttack explanation attack. We found no reported result on Lacuna for its cross-seed agreement.

Full 10-step PGD at the standard $\epsilon$ is the high-compute ceiling. The key ablation is Gaussian noise augmentation at matched perturbation norm. It isolates whether the adversarial direction matters, or whether any smoothing of the input distribution suffices.

A method "beats" the ensemble if two conditions hold. It must reach equal or higher cross-seed RBO at equal training compute. It must also keep clean accuracy within 1 point and keep explanation fidelity no worse. Fidelity is measured by deletion curves and checked against an untrained-model reference, because of the paradox Slack et al. describe, in which untrained models can score better than trained ones on fidelity metrics.

Why not just…?

Why not just raise the Monte Carlo sample count in the explainer? Larger sampling budgets shrink explainer noise but leave model-induced multiplicity untouched (Explanation Multiplicity in SHAP). Our dual-seed design reports both components separately, so this alternative is measured rather than dismissed. We expect the seed component to remain unchanged as the sample count grows.

Why not just average several standard models? This is the real threat, and it is why the ensemble is our main baseline. If compute-matched ensembles match low-dose adversarial training on agreement, our training-side claim fails for anyone who can afford $K$-fold explanation cost at test time. The claim survives only as a test-time efficiency result, and we will say so.

Why not just use noise augmentation or weight decay? Weight decay and $\ell_1$ penalties are already reported to give worse sparsity–accuracy trade-offs than adversarial training (Concise Explanations; Training for Stable Explanation for Free). Gaussian augmentation has not been compared on seed agreement. If it matches single-step adversarial training at equal norm, we will report that the adversarial direction is unnecessary and that smoothing is the operative cause.

Why not just use R2ET? R2ET may well achieve high agreement, since flattening curvature could also reduce sensitivity to training randomness. It is included at matched tuning budget. If it dominates the curve, the contribution becomes evidence that a known cheap regularizer also fixes retraining multiplicity, which is still useful. We will additionally check its output-side stability, which has not been measured.

Decisive Checks

The datasets are German Credit, ACS Income, Diabetes, and Bank for tabular data with KernelSHAP and LIME, SST-2 and IMDB for text with Integrated Gradients and LIME, and Fashion-MNIST and CIFAR-10 for images with gradient methods.

The central claim holds if three things are true. Single-step adversarial training with small $\epsilon$ must recover at least 80% of the gap in cross-seed RBO between standard training and 10-step PGD, at no more than a quarter of PGD's compute. It must beat the compute-matched ensemble and the Gaussian ablation on at least two of the three data types. It must not reduce Relative Output Stability or fidelity below the standard-training level.

A secondary check repeats the agreement measurement on catastrophically overfit single-step models. If agreement survives the collapse of robust accuracy, that directly shows explanation agreement and robustness come apart in compute. Finally, if the feature-level correlation in the mechanistic test is weak, we will drop the feature-suppression explanation and report agreement gains as an empirical effect only.