Adversarial training makes image classifiers resistant to small, deliberately crafted input perturbations. It works by solving a min-max problem: at every update, an inner attack (usually $K$ steps of projected gradient descent, PGD) searches for the worst nearby input, and the outer step trains on it. That makes each update roughly $(K+1)$ times as expensive as ordinary training. The cost is paid twice when a robust model also has to be small. Current pipelines for robust pruning either prune a robust model and then fine-tune it again with multi-step PGD, or prune throughout a full PGD training run. The pruning step is cheap. The adversarial recovery after it is not.
This note proposes removing most of that recovery cost by moving the work into the pruning step. Curvature-informed pruning removes weights using second-order information about the loss, meaning how sharply the loss rises when a weight is changed, and then adjusts the surviving weights in closed form to compensate. Today those curvature statistics and compensation updates are computed on a few hundred clean calibration inputs. Our central claim is that if they are computed on a small set of adversarially perturbed calibration inputs, and the compensation is fit to reproduce the dense robust model's internal responses to those attacks, then a one-shot pruned robust model can keep most of its AutoAttack robustness with little or no adversarial fine-tuning. The total compute needed to produce a robust sparse model would then drop from tens of PGD epochs to minutes of calibration.
Core Claim and What Is New
Three bodies of work frame the gap. Adversarial Pruning: A Survey and Benchmark re-evaluates 11 robust pruning methods under AutoAttack. It finds that the choice of layer-wise sparsity allocation (HARP, LAMP) matters more than the sophistication of the per-weight criterion, so plain magnitude pruning with a good allocator is hard to beat. Every method it evaluates, however, either assumes a robust dense checkpoint followed by adversarial fine-tuning or embeds PGD inside training. None uses second-order curvature, and none treats the adversarial training cost as the quantity to reduce.
Second-order one-shot pruning supplies our method. WoodFisher inverts a block-diagonal empirical Fisher matrix with the Woodbury identity, using about 400 calibration images. SparseGPT recasts pruning as layer-wise reconstruction of the dense layer's outputs and shares one Hessian, $2XX^T$, across all rows of a weight matrix. Both calibrate only on clean data and evaluate only clean accuracy or perplexity.
Input-space curvature work motivates why clean calibration should fail. Understanding and Improving Fast Adversarial Training and Low Curvature Activations Reduce Overfitting in Adversarial Training show that robustness is tied to how the loss behaves inside the perturbation ball around each input, a region clean calibration never visits. Hessian-Free Second-Order Adversarial Examples uses second-order input curvature to speed up the attack itself, but it never touches the weights.
What is new is therefore not "second-order pruning of robust models" in the abstract. It is a specific substitution: adversarially calibrated layer-wise reconstruction is offered as a replacement for adversarial fine-tuning. Its success is measured by AutoAttack robustness per unit of total compute, against the fine-tuning pipelines it aims to replace.
Method
Start from a dense robust network $f$, for example a RobustBench checkpoint (RobustBench). Draw a calibration set of a few hundred to a few thousand training images. Attack each one with PGD against $f$. For each layer with weight matrix $W$, let $X_c$ and $X_a$ be that layer's inputs on the clean and attacked calibration images. The pruned weights $\widehat{W}$ are chosen to solve
$$\min_{\widehat{W}\ \text{sparse}} \; \lambda \,\|W X_a - \widehat{W} \tilde X_a\|_2^2 + (1-\lambda)\,\|W X_c - \widehat{W} \tilde X_c\|_2^2 .$$
Here $\tilde X$ denotes the inputs the layer receives once earlier layers have already been pruned, which is how SparseGPT handles errors that compound across layers. The weight $\lambda \in [0,1]$ trades robust against clean fidelity. Because this objective is quadratic, its Hessian is simply the mixed covariance $2(\lambda \tilde X_a \tilde X_a^T + (1-\lambda)\tilde X_c \tilde X_c^T)$. SparseGPT's row-shared inversion and its closed-form compensation therefore apply unchanged. Layer allocation comes from the resulting reconstruction errors, so it is global and needs no per-layer search.
One subtlety from the research file deserves attention. Classical second-order pruning assumes the gradient of the loss is near zero, and that assumption is shaky at adversarially trained checkpoints, where gradients keep bouncing (On the Loss Landscape of Adversarial Training). The layer-wise reconstruction objective sidesteps this. It is exactly minimized at $\widehat W = W$, so its own gradient vanishes there no matter how far the global adversarial loss is from stationary. We therefore expect reconstruction to beat a global adversarial-Fisher variant of WoodFisher that drops the first-order term. We will test this directly by running the global variant with and without the term $g^T\delta w$, where $g$ is the gradient of the loss with respect to the weights and $\delta w$ is the change the pruning step makes to them.
Attacks generated against the dense model may not be the worst attacks against the pruned one. The method therefore optionally runs $R$ rounds: prune to an intermediate sparsity, regenerate PGD attacks on the partially pruned network, and re-reconstruct. All attack generation counts toward the reported compute. To get real wall-clock gains on GPUs, and not just theoretical ones, every experiment is run both with unstructured sparsity and with the 2:4 semi-structured pattern that SparseGPT supports. Piras et al. find that fully structured filter pruning destroys robustness, so 2:4 is the most hardware-friendly pattern worth testing.
Research Program
The work has three stages. First, establish whether clean-calibrated second-order pruning loses robustness specifically, not just accuracy. We will prune robust checkpoints with clean-calibrated SparseGPT and WoodFisher at 50–95% sparsity and track AutoAttack accuracy separately from clean accuracy. If robustness is preserved as well as clean accuracy, the premise fails and the project shrinks to a benchmarking note.
Second, run the adversarially calibrated method across sparsity levels, values of $\lambda$, calibration sizes, and rounds $R$. For each configuration we record AutoAttack accuracy, clean accuracy, and the GPU-hours from the robust checkpoint to the final sparse model.
Third, measure the residual fine-tuning needed. For each configuration we find the smallest budget of short single-step adversarial fine-tuning that closes the gap to the best fine-tuned baseline. This gives a compute-versus-robustness curve, not a single point.
A secondary analysis addresses an open question in the file: whether pruning along low-curvature weight directions changes how linear the loss is inside the perturbation ball. We measure the gradient-alignment statistic from Understanding and Improving Fast Adversarial Training, the cosine similarity between input gradients at an image and at a nearby perturbed point. We compute it before and after pruning, and during any single-step fine-tuning, to see whether adversarially calibrated pruning makes catastrophic overfitting more or less likely. Catastrophic overfitting is the sudden collapse of multi-step robustness during single-step training.
Baselines
The naive reference point is the dense robust checkpoint itself, together with magnitude pruning at the target sparsity using the same layer allocation and no fine-tuning. This brackets what is achievable for free.
The strongest existing competitor is magnitude pruning with HARP or LAMP layer allocation followed by full multi-step PGD adversarial fine-tuning. This is the A-pipeline that Piras et al. identify as dominant. They report over 40% AutoAttack robust accuracy on CIFAR-10 even at 99% sparsity, and a 2–3% advantage for HARP at high sparsity on ImageNet. No reported result was found on Lacuna for the exact AutoAttack accuracy of this pipeline at intermediate sparsities on our chosen checkpoints, so we will reproduce it with their protocol. It runs on the same checkpoints and data, with the fine-tuning schedule they prescribe. Beating it means one of two things: matching its AutoAttack accuracy within one point at a tenth of its total GPU-hours or less, or exceeding it at equal compute.
The cheap-fine-tuning competitor is magnitude plus HARP followed by FGSM fine-tuning with the GradAlign regularizer. FGSM is a one-step attack. GradAlign penalizes low gradient alignment and matches PGD robustness at $\epsilon = 16/255$ on CIFAR-10 (Understanding and Improving Fast Adversarial Training). This baseline gets exactly the same GPU-hour budget as our method's calibration plus any residual fine-tuning.
Two ablations isolate the key ingredient. The first is the same reconstruction calibrated only on clean data ($\lambda = 0$, i.e. SparseGPT as published), which isolates the effect of adversarial calibration. The second is adversarially calibrated reconstruction with a magnitude-selected mask but the closed-form compensation retained, which separates "which weights go" from "how the survivors are corrected." The global adversarial-Fisher variant, with and without the first-order term, tests the stationarity argument. All methods share the same calibration images, attack settings, and tuning budget for their hyperparameters.
Why not just…?
Why not just use magnitude pruning with a good layer allocator, as the benchmark recommends? This is the most serious threat. Piras et al. find that elaborate criteria lose to magnitude pruning plus HARP. Their comparison, though, is always after adversarial fine-tuning, which can repair a poor mask. Our question is what happens when that repair is removed or cut short. In that regime, compensation updates should matter more, as SparseGPT found when magnitude pruning collapsed beyond 30% sparsity without retraining. The mask-only ablation settles this. If magnitude plus HARP with compensation matches our full method, the curvature-based selection adds nothing and we will report that.
Why not just calibrate SparseGPT on clean data as usual? Clean calibration never visits the perturbation ball, so it has no reason to protect the directions attacks exploit. This is an empirical claim, and it is decided by stage one and the $\lambda = 0$ ablation. If clean calibration preserves AutoAttack accuracy as well as adversarial calibration does, the core claim is false.
Why not just fine-tune briefly with fast adversarial training after pruning? FGSM with GradAlign is cheap and well validated on dense models, and a few epochs of it may recover most robustness on CIFAR-10. We treat it as a compute-matched competitor, not a strawman. We also expect the two to combine: adversarially calibrated pruning should leave less to recover, which shortens the fast fine-tuning. The compute-versus-robustness curves show whether our method shifts the whole frontier or only its zero-fine-tuning endpoint.
Why not just switch to a low-curvature activation? Low Curvature Activations cuts the robust generalization gap on CIFAR-10 from 33.21% to 7.24%. That swap, however, compresses nothing and requires retraining from scratch. It is complementary, and we will include one SiLU checkpoint to check that our gains persist on it.
Proposed Experiments
The main testbed is CIFAR-10 at $\ell_\infty$, $\epsilon = 8/255$, using a ResNet-18 and a WideResNet robust checkpoint, with CIFAR-100 to check that gains hold as label complexity rises. To test scaling we use an adversarially trained ResNet-50 on ImageNet, and a robust ViT. The ViT matters because Global Vision Transformer Pruning with Hessian-Aware Saliency suggests magnitude pruning should be weakest there.
Robustness is always measured with full AutoAttack under RobustBench rules. This matters because both RobustBench and Piras et al. document overestimation of up to 10% and 6.37% respectively when papers rely on PGD alone. Compute is reported as total GPU-hours from the robust checkpoint, including the attacks used for calibration. Wall-clock inference speed is reported for the 2:4 models.
The decisive check is at 90% unstructured sparsity and at 2:4 on CIFAR-10. Adversarially calibrated reconstruction with no fine-tuning should land within two AutoAttack points of magnitude plus HARP with full PGD fine-tuning, at under 5% of its compute. It should also clearly beat the clean-calibrated ablation.
A second check guards against a false success. Adversarial calibration might merely mask gradients, making the model look robust because attacks fail to find good directions. If so, the black-box Square attack inside AutoAttack will reveal a large gap relative to the gradient-based attacks, and the gradient-alignment statistic will collapse. Either outcome would invalidate the result.
References
- Adversarial pruning: A survey and benchmark of pruning methods for adversarial robustness
- WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
- Understanding and Improving Fast Adversarial Training
- Low Curvature Activations Reduce Overfitting in Adversarial Training
- Hessian-Free Second-Order Adversarial Examples for Adversarial Learning
- RobustBench: a standardized adversarial robustness benchmark
- On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
- Global Vision Transformer Pruning with Hessian-Aware Saliency