Calibrated Multi-Test Gates for Training-Free Adversarial Defense

Adversarial training is the standard way to make an image classifier robust to small, deliberately crafted input perturbations. It is expensive, and it lowers accuracy on ordinary inputs. A cheaper alternative defends the model only at test time. It looks at several augmented copies of an input (crops, blurs, frequency filters) and flags the input as adversarial when those copies disagree in a suspicious way. Methods of this kind for CLIP, a vision-language model that classifies images by comparing them to text descriptions, report large robustness gains with no retraining. Their weak point is the cutoff that decides "this input is suspicious." That cutoff is currently a hand-picked number. Nobody knows how often it wrongly rejects clean images. And it is exactly the quantity an attacker who knows the defense will optimize against.

This proposal treats the detection cutoff as the object of study. We calibrate it with a finite-sample guarantee on the clean false-rejection rate, and we evaluate it against attackers who try to stay under it. The central design choice is to split the augmentations into separate families (spatial, pixel, frequency) and calibrate one significance test per family. The tests are then combined with a multiple-testing rule, so an input passes only if it looks clean to every family at once. The hypothesis is that, under a fixed perturbation budget, an attacker must make its perturbation stable under each family separately to pass. That costs the attacker more than evading one pooled score, while the clean rejection rate stays exactly at the level the user chose.

Core Claim

For training-free test-time defenses, the right evaluation object is a curve. On one axis is a guaranteed clean false-rejection rate $\alpha$. On the other is the success rate of a fully adaptive attacker that knows the gate and its threshold. We claim two things. First, replacing heuristic gates with per-family conformal p-values combined by a Benjamini-Hochberg-style rule gives realized clean false-rejection rates that match $\alpha$, even with a few hundred calibration images per dataset. Second, at matched $\alpha$ and matched augmentation count, the multi-test gate forces adaptive attackers to spend more perturbation budget than a single pooled consistency score. This holds under the joint-loss evasion attacks that broke earlier detectors.

The second claim is the risky one, and the experiments are built around falsifying it. If a pooled score with the same augmentations performs as well, the multi-test design adds nothing. The contribution then shrinks to the calibration-plus-adaptive-evaluation protocol, and we will report it as such.

Grounding in Prior Work

Three training-free defenses motivate the setting.

  • Directional Bias-Guided Test-time Defense (DBD) applies 31 spatial, pixel and frequency transformations to CLIP inputs. It scores how concentrated the resulting feature shifts are and corrects the features along that direction. It reports 93.8% average robust accuracy under PGD-100 on fine-grained datasets at 0.025 s/image. It also concedes two weaknesses: the gate threshold is heuristic, and full adaptive attacks combining BPDA (a differentiable stand-in for non-differentiable steps) with EOT (averaging gradients over random transformations) drop accuracy significantly, with about 81% still detected.
  • ATAC corrects CLIP embeddings using augmentation drift vectors behind a cosine-consistency gate. It reports gains of nearly 50% over prior test-time defenses across 13 datasets, again without formal control of its gate.
  • Robustness Tokens takes a parameter-efficient route. It trains 10 input tokens for 200 steps on a frozen backbone, recovering DINOv2-G from 0.3% to 63.1% under white-box attack while clean accuracy moves only from 85.6% to 85.2%. It has no rejection option and relies partly on the secrecy of its tokens.

We keep the zero-training property of DBD and ATAC. What we change is how the gate is set and how it is attacked.

Two results constrain the design.

  • Adversarial Examples Are Not Easily Detected showed that ten detectors fall to attackers who add the detector's score to their loss, often with about 10% more distortion. Any gate we propose must therefore be attacked this way, with the threshold known to the attacker.
  • Indicators of Attack Failure showed that apparent robustness frequently comes from broken attack optimization. Its codebase becomes a required sanity check on every adaptive attack we run.

Three papers supply the calibration machinery.

- **A Closer Look at Generalized BH** frames out-of-distribution detection as multiple hypothesis testing. It shows that detection power conditioned on the calibration set follows a Beta distribution that is unstable for small calibration sets, and that ensemble partitioning (eg-BH) stabilizes it. We reuse this technique, applied to adversarial rather than out-of-distribution inputs, which that paper never evaluated.

Finally, Adversarial Training with Rectified Rejection is the closest conceptual ancestor. It couples two confidence thresholds so that an attacker cannot satisfy both. Its thresholds are tuned empirically at 95% clean acceptance, and it needs an adversarially trained backbone. Our proposal takes the coupling idea but makes each coupled test calibrated and training-free.

Research Program

The defense has three parts.

Families and scores. We group DBD-style transformations into $K$ families, with $K=3$ (spatial, pixel, frequency) as the default. For each family $k$ and test input $x$, we compute a consistency score $s_k(x)$. The score measures how strongly CLIP features of the augmented views drift in a common direction away from the raw view. It is the same statistic DBD and ATAC already compute, just restricted to one family.

Calibration. On a held-out set of $n$ clean images from the deployment dataset, we compute each family's scores. The conformal p-value for a test input is

$$p_k(x) = \frac{1 + \#\{i \le n : s_k(x_i) \ge s_k(x)\}}{n+1}.$$

Here $x_i$ are the clean calibration images. By exchangeability, this p-value is valid for a clean test image: it is no smaller than uniform.

Gate and correction. The $K$ p-values are combined with a Benjamini-Hochberg-type rule at level $\alpha$, with eg-BH partitioning when $n$ is small. The input is rejected if any family's hypothesis is rejected under that rule. Accepted inputs go through the unmodified CLIP prediction. Rejected inputs are either abstained on or routed to DBD's directional correction. We report both options.

Adaptive attack. Each p-value is a step function of its score, so the attacker needs something differentiable to optimize. It gets smooth surrogates of all $K$ scores and the exact thresholds implied by $\alpha$ and the calibration set. It then maximizes misclassification subject to every family score staying below its threshold, using penalized PGD with BPDA+EOT through the augmentations. We also run the selective and orthogonal projected-gradient variants from RADAR, and every run is checked with the attack-failure indicators.

Analysis. The main theoretical question is simple and empirical. For fixed $\ell_\infty$ budget $\epsilon$, how does the minimum budget an attacker needs grow as families are added at fixed $\alpha$? We do not expect a clean bound. We do expect a measurable increase if the families capture genuinely different invariances.

Baselines

Undefended CLIP. The do-nothing reference is zero-shot CLIP ViT-B/16 with no gate. DBD reports 0.0% average robust accuracy for it under PGD-100 on fine-grained datasets. Any defense must also be compared with its clean accuracy, which DBD reports as 67.9% averaged over those datasets.

DBD and ATAC as published. These are the strongest training-free competitors and the direct target of the proposal. They use the same augmentations and the same backbone. The strongest reported number is DBD's 93.8% average robust accuracy under PGD-100, with about 81% of adaptive BPDA+EOT attacks still detected by its DB-score.

For a fair comparison we run both codebases on the same 13 ATAC datasets with the same augmentation pool and the same number of forward passes per image. We then re-set their heuristic gates so that their empirical clean rejection rate equals our $\alpha$ on the same calibration data. Without this step, any advantage could come merely from a different operating point. Both are then attacked with the identical adaptive attack, adjusted to their gate statistic.

Beating them means one of two outcomes at matched realized clean false-rejection rate: lower adaptive attack success (accepted and misclassified), or a larger minimum $\epsilon$ needed to reach a given success rate. Recovering 93.8% under non-adaptive PGD is not the criterion.

Rectified Rejection. This is the strongest method that couples thresholds, though it requires adversarial training. On CIFAR-10 it reports roughly 58% TPR-95 accuracy under PGD-100. We run our gate at $\alpha=5\%$ on a RobustBench CIFAR-10 model and on HAT-trained ResNet-18 models from the HAT codebase. We report accuracy with 95% of clean images accepted, under PGD-100 and under our adaptive attack.

Exceeding 58% without the auxiliary training would be a clear win. Matching it at a small fraction of its training compute would be a qualified win.

Robustness Tokens. This is the parameter-efficient alternative, trained for its published 200 steps on the same backbone. It shows whether a small amount of training beats calibration alone.

Pooled-score ablation. This isolates the key ingredient. It uses the same augmentations and the same forward passes, but pools all views into one consistency score calibrated with one conformal threshold at level $\alpha$. We also run a second pooled variant that takes the maximum of per-family z-scores.

If either pooled variant matches the multi-test gate on adaptive success at every $\alpha$, the multiple-testing structure is not doing the work.

Why not just…?

Why not just take the 95th percentile of the gate score on clean validation data? For a single score, that is essentially split conformal calibration, and we use it as the pooled ablation. It fails in two concrete places.

  • With small per-dataset calibration sets, the achieved detection rate is Beta-distributed and highly variable, as shown in A Closer Look at Generalized BH. We measure realized false-rejection variance at $n \in \{50, 200, 1000\}$ across 100 calibration draws.
  • A percentile rule carries no mechanism that forces the attacker to satisfy several independent constraints.

If the pooled percentile gate is as hard to evade as the multi-test gate, this alternative wins, and we will say so.

Why not just adversarially train and fix confidence with temperature scaling? For CLIP-scale models this reintroduces the retraining cost we are trying to avoid, and it lowers clean accuracy. On calibrated outputs specifically, The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks finds that TRADES-trained models produce very large conformal sets under attack: an average of 22.82 on Caltech-256 at 90% coverage, which drops to 15.03 only with extra uncertainty-reducing training. The CIFAR-10 comparison against adversarially trained RobustBench models lets readers see the accuracy-versus-compute trade-off directly.

Why not just train a separate detector? Plain secondary detectors are bypassed by joint-loss attacks, per Adversarial Examples Are Not Easily Detected. The resilient version, RADAR, reports ROC-AUC near 0.99–1.00 and attack success of 0.00 at 5% false-positive rate, but at a training cost it describes as higher than standard adversarial training.

RADAR is a genuine threat on accuracy. Our claim against it is about cost and guarantees, not about beating its detection numbers. We report GPU-hours alongside any comparison, and we test whether its fixed operating point holds its false-positive rate when calibration sets shrink.

Proposed Experiments

Benchmarks and models. The main suite is CLIP ViT-B/16 zero-shot on the 13 ATAC datasets: CIFAR-10/100, STL-10, Caltech-101/256, OxfordPets, Flowers-102, FGVC Aircraft, StanfordCars, Country211, Food-101, EuroSAT and DTD. Each dataset gets its own calibration split. The CIFAR-10 comparison with Rectified Rejection uses RobustBench and HAT models.

Metrics. At $\alpha \in \{1\%, 5\%, 10\%\}$ we report:

  • realized clean false-rejection rate, with its variance across calibration draws;
  • selective clean accuracy on accepted images;
  • adaptive attack success, defined as the fraction of attacked images that are accepted and misclassified;
  • the minimum $\epsilon$ needed to reach 50% adaptive success;
  • forward passes and wall-clock time per image, compared with published training costs.

Decisive checks. The main claim fails if any of the following happens.

  • Validity: the realized false-rejection rate exceeds $\alpha$ beyond sampling error.
  • Ingredient: the pooled-score ablation matches the multi-test gate's adaptive success curve at equal forward passes.
  • Attack sanity: any reported robustness coincides with triggered attack-failure indicators, which would mean the attack optimization broke rather than the defense held.

Two stress tests follow.

  • Calibration contamination: we inject 1–5% adaptively crafted, gate-evading adversarial images into the calibration set and measure how far thresholds and power shift. This is the adversarial version of the contamination effect described by Robust Conformal Outlier Detection.
  • Family ablation: we vary $K$ from 1 to 5 at fixed total augmentations. This tests whether attacker cost grows with the number of distinct invariances rather than with raw augmentation count.