Adversarial training makes image classifiers resistant to small, worst-case input changes by training them on attacked images, and it usually costs several points of accuracy on ordinary, unattacked images. One persuasive explanation is that many training-time attacks push images in directions that natural images never vary along, and forcing the classifier to resist those directions distorts its decision boundary for no benefit. We propose to measure, for every training attack and on the fly, how natural its direction is. The measuring tool is a learned sparse precision matrix: the inverse of a covariance matrix, with nonzeros only between neighboring pixels. It is predicted per image by a small network trained on clean data. Attacks that point into directions the image naturally varies along get the full perturbation budget. Attacks that point into unnatural directions get a reduced budget.
The bet is that a per-image, edge-aware model of how pixels co-vary can tell useful attacks from harmful ones better than a fixed frequency filter or a foreground mask. It should also do so more cheaply and flexibly than a stored local-PCA tangent space, because the precision matrix is only needed during training and costs a sparse matrix-vector product per example. If the bet holds, it closes part of the clean-accuracy gap without changing the threat model or the deployed classifier, so robust accuracy can be checked with standard attacks.
Core Claim
A Gaussian Markov random field (GMRF) is a Gaussian model whose precision matrix $Q$ is sparse. Here, $Q_{ij}=0$ means pixels $i$ and $j$ are independent once their neighbors are known. Two lines of work make this usable. Dorta et al. supplies the method: a CNN predicts a sparse lower-triangular Cholesky factor $L(x)$ over a $5\times5$ neighborhood, so $Q(x)=L(x)L(x)^\top$ is always valid. Their CelebA results show the predicted structure separates noise from genuine texture (denoising MSE $2.99\times10^{-3}$ vs. $5.13\times10^{-3}$ for a denoising autoencoder). Simpson et al. show the same parameterization runs in a single forward pass with near-linear cost.
The motivation comes from Yi et al. (TART), who estimate local tangent spaces of the data with an autoencoder plus PCA. They find that perturbations orthogonal to those tangent spaces drive clean-accuracy loss. Restricting full-budget training to tangent-aligned perturbations recovers clean accuracy without losing robustness. Rade and Moosavi-Dezfooli give a complementary diagnosis: adversarial training pushes boundaries much farther from clean points than the threat radius requires, an "excessive margin."
Our claim is that a sparse, input-conditioned precision matrix is a better and cheaper detector of "unnatural" perturbation directions than TART's dense local PCA. For a training attack $\delta$ on image $x$ with $N$ pixel values, define the naturalness ratio
$$r(\delta;x)=\frac{\delta^\top Q(x)\,\delta \,/\, \delta^\top\delta}{\operatorname{tr}Q(x)/N},$$
which is the precision of the attack direction relative to the image's average precision. The quantity $\delta^\top Q\delta=\|L^\top\delta\|^2$ costs one 25-tap sparse product per channel. Values of $r$ below 1 mean $\delta$ points along directions with more natural variance than average, for example along edges and textures. Values above 1 mean it fights the image's correlation structure. The training budget becomes
$$\epsilon_{\text{eff}}(x)=\epsilon\cdot\sigma\!\big(-(\log r(\delta;x)-b)/T\big),$$
where $\sigma$ is the logistic function, and $b$ and $T$ are a threshold and temperature tuned on held-out data. The deployed classifier is an ordinary network evaluated under the standard $\ell_\infty$ ball of radius $\epsilon$.
Research Program
The work has three stages.
First, we train a precision-predicting network on clean CIFAR-10, CIFAR-100, SVHN and TinyImageNet-200 training images. It follows the residual formulation of Dorta et al.: an encoder-decoder predicts a mean and a sparse Cholesky factor over a $5\times5$ neighborhood with cross-channel terms, trained by Gaussian likelihood. We validate it on held-out likelihood and on whether $r$ ranks perturbations sensibly: Gaussian noise should score high, and small natural transformations should score low.
Second, we run precision-gated adversarial training inside standard adversarial training, TRADES (Zhang et al.) and MART. We compute $L(x)$ once per minibatch, on the augmented image actually being trained on. A first PGD pass produces $\delta$ and $r$, and a second pass runs at $\epsilon_{\text{eff}}$, mirroring TART's adaptive-budget protocol so the comparison isolates the direction detector.
Third, we analyze mechanism. We correlate $r$ with TART's tangent-alignment score on the same perturbations. Following Rade and Moosavi-Dezfooli, we measure boundary distance along natural (low-$r$) and unnatural (high-$r$) directions. This tests whether gating removes excess margin specifically along unnatural directions.
Because $Q$ never touches inference, the method cannot create gradient masking at test time. That is the failure mode Croce et al. showed inflated many earlier trade-off claims. We nonetheless evaluate everything with AutoAttack.
Baselines
The naive reference point is standard PGD adversarial training at $\epsilon=8/255$ with the same architecture, epochs and augmentation. For the trade-off curve, it is TRADES (Zhang et al.) swept over its weight $\lambda$, which that paper reports moving smoothly between 88.64% natural and 56.61% PGD robust accuracy on CIFAR-10. We compare Pareto curves, not single points, so a method that merely sits at a different $\lambda$ does not count as winning.
The strongest competitor is TART (Yi et al.), because it uses the same adaptive-budget mechanism with a different direction detector. On WideResNet-32-10 CIFAR-10 it reports clean accuracy rising from 81.76% to 83.73% over standard adversarial training, with AutoAttack robustness moving from 35.89% to 36.58%. Combined with MART, it reports clean accuracy from 80.18% to 84.31%, at about 1.24× the cost of standard adversarial training. We re-run TART with its released protocol on the same architecture and epochs, with the same hyperparameter search budget for its thresholds as for our $b$ and $T$. Beating it means a higher clean accuracy at equal or better AutoAttack accuracy, beyond seed variance across three seeds, or equal results at lower wall-clock and memory cost with dynamic augmentation enabled.
We also run HAT (Rade and Moosavi-Dezfooli), reported at about +2.17% clean over TRADES with roughly 49% AutoAttack accuracy on PreAct ResNet-18. We run PART (Improving Accuracy-robustness Trade-off via Pixel Reweighted Adversarial Training), reported at +1.2% to 3.18% clean accuracy. Both are orthogonal enough that we also test their combination with gating.
Three ablations isolate the key ingredient, the input-conditioned sparse off-diagonal structure of $Q$:
- Stationary GMRF. A single image-independent $Q$, a fitted lattice precision diagonalized by FFT as in Sahu et al.. This is in effect a frequency filter.
- Diagonal $Q(x)$. Per-pixel variance only, a learned soft pixel reweighting.
- Random gating. Budgets drawn at random with the same distribution of $\epsilon_{\text{eff}}$.
Why not just…?
Why not just tune TRADES or use HAT? These shift or modestly improve the frontier without asking which perturbations hurt clean accuracy, and HAT needs a separately trained teacher. Our evidence is the Pareto comparison. If gated training lies on the TRADES curve, we have only reparameterized $\lambda$. Combining gating with HAT tests whether the two remove different parts of the excess margin.
Why not just low-pass the perturbations with a fixed smoothness prior? This is the most serious threat. Sahu et al. show that attack gradients are spatially correlated and well described by a stationary lattice GMRF. Much of what $r$ captures could therefore be plain frequency content. The stationary-GMRF ablation settles this directly. If it matches the input-conditioned version, the honest conclusion is that frequency, not image-specific structure, drives clean degradation, and we would report the cheaper method. Our prior that it will fall short rests on Dorta et al.: image-conditioned precision preserves edges and textures that stationary smoothing discards, and these are exactly the high-frequency directions natural images do vary along.
Why not just use TART's local PCA? TART works, so this is a genuine competitor, not a strawman. Its stated limitations are concrete. It stores precomputed tangent bases per training example and cannot follow augmentations applied on the fly. A dense PCA basis in autoencoder latent space describes marginal variance, not which pixels depend on which. Our per-image $Q$ is recomputed on whatever augmented image is in the batch. We test the difference in two ways: head-to-head under TART's own setting, and with strong dynamic augmentation, where TART must fall back to stale tangents.
Why not just purify inputs with a diffusion model? MimicDiffusion (Song et al.) and SSNI (Sun et al.) report strong clean and robust numbers. They change the deployed system and add test-time cost; SSNI reports 0.5–5 s per image. They also introduce the iterative inference that Croce et al. warn complicates robustness evaluation. We treat them as a different deployment regime and report their published numbers for context only.
Proposed Experiments
Main comparison. The primary experiment is CIFAR-10 at $\ell_\infty$ $\epsilon=8/255$ on PreAct ResNet-18 and WideResNet-32-10. We report clean accuracy, PGD-20 accuracy and AutoAttack accuracy via RobustBench, over three seeds. All baselines and ablations get matched epochs and tuning budgets. We repeat on CIFAR-100, SVHN and TinyImageNet-200, and report CIFAR-10-C accuracy, since unnatural-direction training may also affect corruption robustness.
Decisive check 1: is the gain real? Gated training must beat TART's clean accuracy at no AutoAttack loss on WideResNet-32-10.
Decisive check 2: does conditioning matter? The input-conditioned $Q$ must beat the stationary and diagonal ablations. Otherwise the sparse conditional structure is not what matters.
Decisive check 3: does flexibility pay? Under dynamic augmentation, the gap over TART should widen.
Mechanism checks. Boundary-distance measurements should show the clean-accuracy gain coinciding with reduced margin along high-$r$ directions only. Correlating $r$ with TART's alignment score tells us whether the two detectors agree, or whether $Q$ flags different perturbations. We also report wall-clock and memory overhead relative to standard adversarial training, against TART's 1.24×.
References
- Structured Uncertainty Prediction Networks
- Learning Structured Gaussians to Approximate Deep Ensembles
- TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training
- Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off
- Theoretically Principled Trade-off between Robustness and Accuracy
- RobustBench: a standardized adversarial robustness benchmark
- Improving Accuracy-robustness Trade-off via Pixel Reweighted Adversarial Training
- Gaussian MRF Covariance Modeling for Efficient Black-Box Adversarial Attacks
- MimicDiffusion: Purifying Adversarial Perturbation via Mimicking Clean Diffusion Model
- Sample-specific Noise Injection for Diffusion-based Adversarial Purification