Paying for Robustness at Deployment: Label-Free Adversarial Tuning of Normalization Parameters on the Test Stream

Adversarial training makes a classifier resistant to tiny, deliberately crafted input perturbations. It does so by training on the worst perturbation found by an inner optimization loop. That loop multiplies training cost by roughly an order of magnitude, and the cost has to be paid in full before deployment, on labeled data. Test-time adaptation follows a different logic: a deployed model keeps adjusting a tiny set of parameters on the unlabeled data it actually sees. The usual targets are the per-channel scale and shift ("affine") parameters of its normalization layers, fewer than 1% of all weights. This note asks whether the first job can be moved onto the second mechanism. Concretely, can a standard, non-robust pretrained model acquire a meaningful fraction of adversarial-training robustness by running a cheap, label-free robustness objective on its normalization parameters as test batches stream in, with the cost spread across deployment instead of paid up front?

The opportunity matters because the two directions currently fail in complementary ways. Static adversarial training is expensive and brittle under natural shift. Test-time adaptation handles natural shift but is itself easy to poison and gives no protection against per-sample attacks. Earlier attempts to defend at test time were mostly shown to be illusory once the attacker differentiated through the defense. The design proposed here is built so that its robustness can be checked with ordinary, unmodified attacks.

Core Claim

The objective is a label-free version of the clean-versus-perturbed agreement term used in adversarial training. Applied only to normalization affine parameters $\gamma, \beta$ and driven by a single perturbation step per test batch, it can turn a standard model into one whose frozen snapshot is robust under AutoAttack. It can do this while matching entropy-minimization adaptation on natural corruptions, at a per-batch cost of a few forward and backward passes.

Two design choices separate this from the test-time defenses that Evaluating the Adversarial Robustness of Adaptive Test-time Defenses broke. That paper re-attacked nine defenses whose per-input optimization loops hid gradients and found, for example, Shi et al. dropping from 51.02% to 3.7%. First, the adaptation here is slow and stream-level rather than per-input. At any moment the deployed model is a fixed, deterministic network, so it can be attacked with standard white-box attacks without approximating the gradient of an inner optimization loop. Second, the threat that adaptation does introduce, namely poisoning the stream, is evaluated directly with the label-free attacks of On the Adversarial Vulnerability of Label-Free Test-Time Adaptation.

For each test batch, let $p_\theta(x)$ be the model's predicted class distribution with normalization parameters $\theta = (\gamma, \beta)$ and all other weights frozen. The update minimizes

$$\mathcal{L}(\theta) = \frac{1}{B}\sum_{i=1}^{B}\Big[ H\big(p_\theta(x_i)\big) + \lambda\, \mathrm{KL}\big(\mathrm{sg}[p_\theta(x_i)] \,\|\, p_\theta(x_i+\delta_i)\big)\Big] + \mu \|\theta - \theta_0\|^2 .$$

The terms are defined as follows. $B$ is the batch size and $H$ is Shannon entropy, as in Tent. $\mathrm{sg}$ stops gradients through the clean prediction, which serves as the pseudo-target. Each $\delta_i$ is a single-step, randomly initialized perturbation within an $\epsilon$-ball that maximizes the KL term. $\theta_0$ are the source parameters, and $\lambda, \mu$ are weights. The agreement term needs no labels. The entropy term and the pull toward $\theta_0$ are there to block the trivial solution in which the model predicts the same distribution for every input.

Research Program

The method borrows from three sources. From Tent: Fully Test-Time Adaptation by Entropy Minimization it takes the update rule: online, label-free gradient steps on normalization affine parameters. From Reliably fast adversarial training via latent adversarial perturbation it takes a way to make single-step perturbations reliable. SLAT perturbs both inputs and hidden activations in one step, which avoids the collapse that plain single-step training suffers; there, FGSM falls to 0.01% PGD-50 accuracy while SLAT reaches 47.06%. A SLAT-style latent perturbation will be tested as the choice of $\delta$. From MedBN: Robust Test-Time Adaptation against Malicious Test Samples it takes median-based batch statistics, so that a few malicious samples cannot drag the normalization statistics.

Three papers motivate the bet that affine parameters alone carry enough capacity for robustness. FullLoRA-AT found that standard and adversarially trained vision transformers differ substantially in their LayerNorm scale and shift. Perturbing BatchNorm and Only BatchNorm Benefits Sharpness-Aware Minimization found that restricting worst-case parameter perturbations to BN affine parameters beats perturbing all parameters. Revisiting adapters with adversarial training reached 49.87% robust accuracy on ImageNet with only 768 extra parameters, in the form of a second classification token. None of these three work label-free or at test time.

Removing Batch Normalization Boosts Adversarial Training constrains the design in two ways. It shows that BN statistics computed over mixed clean and adversarial batches hurt accuracy, and it notes that BN-free robust models cannot use test-time adaptation. The method therefore computes statistics from clean samples only and applies them to the perturbed copies, so the mixture never forms. This is the sense in which the proposal addresses that paper's dilemma: normalization layers are kept, so adaptation remains possible, without the mixed-batch accuracy loss.

The work has four parts. The first builds the update on BN ResNets (CIFAR-10, ImageNet ResNet-50) and LayerNorm ViT-B. The second measures how robustness of the frozen snapshot grows as a function of the number of test batches seen, with the source domain served as an unlabeled stream. The third repeats this on corrupted streams to see whether robustness and corruption recovery coexist. The fourth attacks the adaptation process itself.

Baselines

The naive reference is the unadapted standard model, alongside test-time BN statistic re-estimation with no gradient steps. Both should give near-zero AutoAttack accuracy. On ImageNet-C, statistic re-estimation alone has a reported error of 49.9% in Tent.

The natural-shift reference is Tent itself, run on the same streams, batch sizes and number of steps. It is reported at 44.0% ImageNet-C error. The proposal must stay within about one point of that error to count as preserving adaptation.

The strongest robustness competitor is static PGD adversarial training. Adaptive Batch Normalization Networks for Adversarial Robustness reports PGD-AT at 51.6% robust and 83.3% clean accuracy on CIFAR-10. We will retrain PGD-AT on the same architecture and evaluate it with AutoAttack through RobustBench conventions. On the ImageNet scale the parameter-efficient upper reference is the dual-token adapter, with 49.87% robust accuracy. Matching PGD-AT is not the bar, because the proposal uses no labels and far less compute. The bar has two parts. The first is to beat the cheaper existing substitutes by a clear margin: ABNN reports 31.5% white-box robust accuracy, and the input-purification defense DAD reports 37.37% under AutoAttack, both on CIFAR-10. The second is to reach these numbers at a reported total FLOP cost well below PGD-AT. On corrupted-and-attacked data, the target is to beat PGD-AT combined with Tent, which is the combination the Distribution Invading Attack paper suggests as a defense.

Supervised parameter-efficient adversarial training (FullLoRA-AT style, but affine-only, with labels and multi-step PGD) is run with the same parameter set. It measures how much robustness the affine parameters can hold at all when labels and full PGD are available. This gives a ceiling for the label-free, single-step version.

The decisive ablation is offline adversarial tuning of the same affine parameters on unlabeled source-domain data before deployment, with the same objective and the same total number of updates. It separates "adversarial tuning of affine parameters" from "doing it on the test stream." Further ablations remove the agreement term (which reduces the method to Tent), swap the latent perturbation for plain FGSM, and replace median statistics with mean statistics.

Why not just…?

Why not just adversarially train the affine parameters offline and deploy a static model? This is the real threat, and the offline ablation is designed to settle it. If offline tuning matches online tuning on clean-domain streams and on shifted streams, then the honest contribution narrows. It becomes "label-free, single-step, affine-only adversarial tuning is a cheap robustness add-on," and the test-time framing helps only through the accompanying Tent term. The hypothesis that online tuning wins rests on the fact that perturbations crafted around corrupted inputs differ from those around clean ones. Whether that matters is an empirical question; the file reports no robust-under-corruption numbers to settle it in advance.

Why not just run Tent on an adversarially trained model? It works, and the Distribution Invading Attack paper (Uncovering Adversarial Risks of Test-Time Adaptation) suggests an adversarially trained base partly resists poisoning. But it pays the full cost of adversarial training up front. It also inherits the clean-accuracy loss that NoFrost reports for BN-based training on ImageNet (about 59% versus 74%). We run it as a baseline and report accuracy against total compute, so the comparison concerns cost as well as peak robustness.

Why not just use robustness tokens or the dual classification token? Robustness Tokens recovers DINOv2-G from 0.3% to 63.1% white-box accuracy in 200 steps, but its security depends on the tokens staying secret. The dual-token adapter needs labeled adversarial training, and neither method adapts to shift after deployment. Our evaluation assumes the attacker knows the adapted parameters at every snapshot, which is a stricter threat model. If tokens still dominate under equal compute on ViT-B, we report that.

Why not just purify inputs at test time? ATAC and DAD avoid touching weights, but DAD stops at 37.37% under AutoAttack on CIFAR-10. Croce et al. show that purification loops tend to fail once the attacker substitutes a differentiable approximation for the purifier. Our snapshot design avoids this failure mode by construction rather than by argument.

Proposed Experiments

The main grid crosses three settings: CIFAR-10/CIFAR-10-C with a ResNet, ImageNet/ImageNet-C with ResNet-50, and ImageNet with ViT-B. Every method is evaluated on the same data at three points of the stream: on the clean stream, on each corruption stream, and on attacked inputs. Attacked inputs are measured as AutoAttack $\ell_\infty$ robust accuracy on frozen snapshots taken after $k$ batches. Compute is reported as total FLOPs and wall-clock time for both offline and online phases.

Four checks decide the outcome. The first is the obfuscation check. Snapshot robustness must hold under full AutoAttack, transfer attacks from PGD-AT models, and a sweep over the number of attack steps. Robustness that disappears as steps increase indicates gradient masking from single-step training, and that would be reported as failure.

The second is the stream-poisoning check. DIA and the label-free Feature Collapse Attack are adapted to target the new objective: the attacker controls 5% to 20% of a batch and aims to erase snapshot robustness or raise benign error. The defense must keep benign error near MedBN's reported clean cost (14.92% to 15.19% on CIFAR-10-C).

The third is the capacity check. If the supervised affine-only ceiling itself sits near ABNN's 31.5%, normalization parameters lack the capacity for robustness on that architecture. The idea would then be restricted to architectures where they do, possibly only LayerNorm transformers, as FullLoRA-AT's findings hint.

The fourth is the online-versus-offline gap on corrupted-and-attacked accuracy. This single comparison determines whether test-time adaptation is the mechanism or merely the setting in which the method runs.