Clean Supports, Robust Queries: Separating Prototype Noise from the Few-Shot Robustness Tax

Few-shot classifiers recognize a new class from one to five labeled examples. The standard design, a prototypical network, averages those examples' features into a single "prototype" per class and labels a new image by its nearest prototype. Making such a classifier robust to adversarial perturbations (tiny, deliberately chosen pixel changes that flip predictions) costs heavily in ordinary accuracy. On 5-way 5-shot mini-ImageNet, a naturally trained model scores 73.01% on clean images and 0.00% under attack. Its adversarially trained counterpart scores 57.87% clean and 31.52% robust (Adversarially Robust Few-Shot Learning: A Meta-Learning Approach). The field has treated this 15-point clean loss as the same robustness-accuracy trade-off seen in standard classification.

We think much of that loss has a different cause. Robust features are less sharply clustered by class, so averaging five of them gives a noisier prototype than averaging five clean features. In the usual few-shot threat model, though, the adversary attacks only the queries (the images to be classified). The labeled support examples are clean and trusted, and nothing forces them through the robust encoder. We propose to measure how much of the robustness cost is prototype noise. We then remove that part with a network that builds prototypes from a clean feature path and sends every query through a robust path trained to land in the same space.

Core Claim

The clean-accuracy penalty of adversarial few-shot learning splits into two parts. One is a feature-quality term that would remain even with perfect prototypes. The other is a prototype-estimation term that exists only because prototypes come from $K$ noisy samples. We claim the second term is large, grows as $K$ shrinks, and can be recovered without giving up robustness. The fix is an asymmetric design: supports are encoded by a clean path, queries by a robust path. Because the adversary perturbs only queries, this asymmetry adds no attack surface.

The decomposition is simple to state. Let $R(\phi, P)$ be the error of nearest-prototype classification using encoder $\phi$ and prototypes $P$. Let $\hat P$ be prototypes from $K$ support samples and $P^*$ be "oracle" prototypes averaged over all of a novel class's images. For a clean encoder $\phi_c$ and a robust encoder $\phi_r$, the clean-accuracy gap is

$$\Delta = \underbrace{R(\phi_r,P^*) - R(\phi_c,P^*)}_{\Delta_{\text{feat}}} + \underbrace{\big[R(\phi_r,\hat P)-R(\phi_r,P^*)\big]-\big[R(\phi_c,\hat P)-R(\phi_c,P^*)\big]}_{\Delta_{\text{proto}}}.$$

The same split applies to the boundary error defined in Theoretically Principled Trade-off between Robustness and Accuracy. That paper bounds robust error by natural error plus the probability of lying within an $\epsilon$-ball of the decision boundary. Prototype noise moves the boundary, so it should inflate boundary error as well as natural error. Prototype variance scales as $\sigma^2/K$, where $\sigma^2$ is the within-class feature variance. We therefore predict $\Delta_{\text{proto}}$ shrinks roughly as $1/K$, while $\Delta_{\text{feat}}$ stays flat in $K$. Robust features should show a lower ratio of between-class to within-class spread, which makes $\Delta_{\text{proto}}$ larger for $\phi_r$.

How This Relates to Prior Work

Three papers motivate the gap. Goldblum et al.'s Adversarial Querying perturbs only the query set during episodic training and keeps supports clean. This is the threat model we adopt, but it still builds prototypes in the same robust features, and the paper reports the 15-point clean drop without diagnosing it. A Simple Approach to Adversarial Robustness in Few-shot Image Classification freezes a PGD-trained backbone and corrects noisy centroids by averaging them with the nearest base-class centroids (Calibrated Nearest Centroid, CNC). It implicitly agrees that prototype noise matters for robustness. However, it reports no clean-accuracy curve, and its correction depends on base and novel classes overlapping semantically. Our fix does not use base centroids at all. Defensive Few-shot Adversarial Learning aligns clean and adversarial local-feature distributions with symmetric divergences. It does not analyze centroid error.

The prototype-instability literature supplies the diagnosis but evaluates clean data only. Prototype Rectification for Few-Shot Learning names the intra-class bias between a small-sample mean and the true center. Its fix, pseudo-labeling and mean-shifting queries, is transductive (it uses the batch of unlabeled queries). That makes it exploitable when queries are adversarial, so it constrains our design: the support estimator must never read query statistics. ProtoDiff refines prototypes with diffusion but is about 15x slower at inference. NAPP suppresses natural noise and was never attacked.

The method comes from Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off (AR-AT). AR-AT pulls adversarial features toward stop-gradient clean targets through a small predictor network. It also keeps separate batch-normalization statistics for clean and adversarial inputs ("Split-BN"), because mixing them corrupts both. On CIFAR-10 this reaches 90.89% clean and 50.77% AutoAttack accuracy. AR-AT uses the two paths only during training, since in standard classification you cannot know at test time which inputs are clean. Few-shot learning is the one setting where part of the test-time input, the support set, is known to be clean. Our contribution is to keep the clean path at test time for supports only.

Research Program

The first stage is the diagnostic. We train matched clean and PGD-adversarial ResNet-12 and WideResNet-28-10 backbones on base classes, following the CNC transfer recipe. On novel classes we evaluate nearest-prototype classification with empirical prototypes for $K \in \{1, 5, 10, 20\}$ and with oracle prototypes from all 600 images per class. Each condition is measured on clean queries and under PGD and AutoAttack at $\ell_\infty$ $\epsilon=8/255$. This yields $\Delta_{\text{feat}}$ and $\Delta_{\text{proto}}$ for both natural and boundary error. We also measure each backbone's between-class to within-class variance ratio to check whether it predicts $\Delta_{\text{proto}}$.

The second stage is the method, which we call the asymmetric prototype network. A single backbone has two normalization tracks (Split-BN).

  • Clean track. It is trained with ordinary episodic cross-entropy on clean images.
  • Robust track and predictor. These are trained, AR-AT style, so that adversarial query features land on the stop-gradient clean-track features of the same image.
  • Training episodes. Supports pass through the clean track. Adversarially perturbed queries, crafted against the full pipeline, pass through the robust track and predictor and are classified against clean-track prototypes.
  • Test time. Supports always use the clean track and every query, clean or not, uses the robust track.

Any clean-accuracy gain must therefore come from better prototypes, not from routing easy queries around the defense. The design has no transductive step, adds one forward pass per support image, and needs no test-time optimization.

The third stage asks whether gains survive where CNC should fail. FC100 builds its splits from disjoint superclasses, and the cross-domain benchmarks (CropDisease, EuroSAT, ISIC, ChestX) share no semantics with mini-ImageNet. CNC's base-centroid averaging has nothing relevant to average there. The clean-support design should be indifferent to this.

Baselines

The naive reference points come from Goldblum et al. on 5-way 5-shot mini-ImageNet under $\ell_\infty$ PGD at $8/255$. Naturally trained R2-D2 scores 73.01% clean and 0.00% robust. Robust transfer learning, a frozen PGD backbone with a standard head, scores 39.13% clean and 25.33% robust. We rerun both in our codebase so every method shares the same backbone, base data, PGD settings, and evaluation episodes.

Adversarial Querying is the strongest published method that reports both numbers in this setting: 57.87% clean and 31.52% robust with R2-D2 (Goldblum et al.). We rerun it with a ProtoNet head on our backbone and give it the same number of attack steps and tuning trials as our method. Beating it means at least 8 points more clean accuracy at robust accuracy within one point of AQ, or higher on both.

CNC is the strongest prototype-correction competitor. It reports 35.40% robust accuracy on 5-shot and 21.38% on 1-shot mini-ImageNet with WideResNet-28-10. No clean accuracy for CNC is reported on Lacuna, so we will measure it ourselves using the authors' recipe, including weight averaging, and tune its number of base neighbors on validation classes. Beating CNC means matching its robust accuracy on mini-ImageNet while exceeding its clean accuracy. On FC100 and cross-domain benchmarks we must exceed it on both.

We also report MDAT's 62.68% adversarial accuracy on 5-shot mini-ImageNet (Defensive Few-shot Adversarial Learning). No clean number for MDAT is reported on Lacuna, so we rerun it if code permits and otherwise cite it as context only.

The key ablation is the same trained asymmetric network with supports routed through the robust track. It shares weights, training, and compute with the full method and differs only in where prototypes are built, so it isolates the claimed mechanism. A second ablation removes the predictor network, to test whether alignment into clean space is necessary.

Why not just…?

Why not just lower the TRADES weight to buy back clean accuracy? Lowering the weight $\lambda$ moves the model along the trade-off curve; it does not move the curve. Our claim is that the curve itself is shifted down by prototype noise. We will sweep $\lambda$ for an episodic TRADES baseline and plot its clean-robust frontier. The method wins only if it lies strictly above that frontier.

Why not just use CNC or shrink prototypes toward the base mean? This is a real threat on mini-ImageNet, where base and novel classes overlap. Shrinkage attacks the same estimation error we identify. Its weakness is that it pulls prototypes toward a biased target, which Subramanya and Pirsiavash themselves expect to fail without semantic overlap. The FC100 and cross-domain comparisons settle this. If CNC matches us on mini-ImageNet but loses under domain shift, the contribution is specific to that regime. If it matches us everywhere, the decomposition remains the result and the method is not needed. We will also test whether the two corrections stack.

Why not just tighten class clusters in the robust encoder with a center-style loss? Shrinking within-class spread would reduce $\Delta_{\text{proto}}$ directly. We run it as a baseline with the same training budget. Our expectation, from AR-AT's finding that mixing clean and adversarial objectives in shared parameters causes gradient conflict, is that tightening robust features costs some robustness. Routing supports through the clean track gets cleaner prototypes without touching the robust track. If the center-style loss matches us on both axes, we will report the simpler method as the answer.

Why not just use transductive rectification on the queries? BD-CSPN-style mean shifts and query pseudo-labels hand the adversary a direct handle on every prototype. We will test this by attacking BD-CSPN on top of AQ features with perturbations crafted to drag the query mean.

Proposed Experiments

The benchmarks are mini-ImageNet, tieredImageNet, CIFAR-FS, FC100, and CUB in 5-way 1-shot and 5-shot. Cross-domain tests use CropDisease, EuroSAT, ISIC, and ChestX with mini-ImageNet as base data. Starting code is the ProtoNet episode tooling in renmengye/few-shot-ssl-public. We report clean accuracy, PGD-20 and AutoAttack robust accuracy, and the harmonic score MDAT introduced for summarizing the trade-off, each over 2,000 test episodes with 95% intervals.

Four checks decide the outcome. First, the decomposition must show $\Delta_{\text{proto}}$ is a substantial share of $\Delta$ and shrinks with $K$. If oracle prototypes on the robust encoder recover little clean accuracy, the hypothesis is wrong and the method has no reason to work. Second, the full method must beat its robust-support ablation in clean accuracy at equal robust accuracy, with the margin largest at 1-shot. Third, robustness must hold under adaptive white-box attacks that differentiate through both tracks, under expectation-over-transformation attacks, and with gradient-free Square attacks; this rules out gradient masking. Fourth, we deliberately break the threat model by poisoning one support image. This tells readers exactly what the clean-support assumption buys and what it costs when it fails.