Keep, Crop, or Drop: Episode-Level Saliency for Few-Shot Rare-Signal Support Sets

When a classifier must learn a rare target from five labeled examples, one wrong example is a fifth of everything it knows. In rare-event settings the two things that go wrong look alike from the outside. Some examples were never signal at all: background events mislabeled as positives. Others are real signal that occupies a small corner of an otherwise cluttered input. Both kinds of example sit far from the class average, so methods that judge each example as a whole cannot tell them apart. They either discard real signal or keep corrupted labels. We propose to let the episode explain itself. For each labeled example, we ask the few-shot classifier which spatial regions push that example toward its stated class and away from the competing classes, using a prototype built from the other examples. The resulting map supports a three-way decision. A clean example is kept whole. A cluttered but valid example is cropped to its supporting region. A mislabeled example, whose map is diffuse or whose cropped embedding still does not return to its class, is dropped.

The motivation comes from three fields that rarely talk to each other. Rare-event searches in physics and imbalanced vision show that background is huge and varied. Few-shot noise research shows that small support sets collapse under corruption. Gradient saliency (maps of which input regions most change a model's output) has mostly been used as a post-hoc check for whether a model attends to the right evidence. Our claim is that saliency computed inside the episode is a cheap and useful denoising signal. It needs no text model and no external saliency network, and it is most valuable when the noise follows the rare-event pattern, with background masquerading as signal.

Core Claim

The robust few-shot methods in the file judge support examples either globally or with unguided crops. Rank-guided Diffusion for Noise Few-Shot Learning (CRDProto) finds noisy support items by how far they deviate from a low-rank subspace and by mismatch with CLIP text embeddings. It then repairs the items with diffusion and reaches 81.39% at 40% label noise on miniImageNet, against 57.07% for a plain prototype network. Its own stated limitation is the reliance on vision-language priors, which do not exist for detector waveforms or specialized imagery. DETA is the only method in the file that explicitly separates image clutter ("X-noise") from wrong labels ("Y-noise"). It does this by weighting random crops by contrastive relevance, which costs many forward passes, and the spatial search is not guided by the classifier. SalNet does use foreground/background masks in few-shot learning, but the masks come from a frozen external saliency model and the labels are assumed clean. These three papers motivate the gap. None of them uses the classifier's own episode-conditioned attribution to decide whether an outlier is cluttered signal or mislabeled background.

The mechanism has a simple closed form, so it should be stated precisely. Take a support example $x$ with stated label $y$. Its backbone feature map has local vectors $A_{ij} \in \mathbb{R}^K$ at spatial positions $(i,j)$, and its pooled embedding $f$ is their mean. Let $c_k^{(-x)}$ be the prototype (mean embedding) of class $k$ computed without $x$. Let $p_k$ be the softmax over negative squared distances from $f$ to each $c_k^{(-x)}$. For the episode log-likelihood $\log p_y$, the gradient with respect to $f$ is proportional to $c_y^{(-x)} - \sum_k p_k c_k^{(-x)}$, and the $f$ terms cancel. Grad-CAM at the last layer then reduces to the following map:

$$M_{ij}(x) = \mathrm{ReLU}\Big(\big\langle A_{ij},\; c_y^{(-x)} - \textstyle\sum_k p_k\, c_k^{(-x)} \big\rangle\Big).$$

The map scores each location by how much it favors the stated class over the probability-weighted competitors. Background that resembles every class cancels out. This contrastive cancellation is what separates the map from simply measuring similarity to the class prototype, and we test that difference directly.

We also adopt the "Top-M" channel filter of Adapting Grad-CAM for Embedding Networks, which keeps only the most strongly weighted channels. That paper reports it raises CUB-200 box localization from 0.643 to 0.760 over unfiltered maps, on clean data only.

Research Program

The method runs in four steps at adaptation time.

First, compute $M(x)$ for every support example from leave-one-out prototypes. Contaminated prototypes can mislead this step, so the maps and prototypes are refined over two alternating passes.

Second, form a cropped embedding $f_M$ by pooling $A_{ij}$ with weights $M_{ij}$.

Third, make the three-way decision from two quantities: how concentrated $M$ is (its spatial entropy), and the margin of $f_M$ for class $y$ under the leave-one-out prototypes. A clean example has a positive whole-image margin and is kept whole. A cluttered valid example has a concentrated map and its cropped margin turns positive, so it is cropped. A mislabeled example has a diffuse map or a cropped margin that stays negative, so it is dropped or given low weight.

Fourth, compute the final prototypes from the kept and cropped embeddings.

There is no second network, no text encoder, and only one backward pass per support example. The map is linear in the last-layer features, so most of its cost is a dot product. We will also run the version computed at an intermediate layer. There the gradient is genuinely nonlinear, and it is an empirical question whether it localizes better than the last layer.

The rare-event direction sets the noise model. In real rare-event searches, label errors are not uniform swaps between classes. A positive that should not be there is almost always a background event. We therefore build two noise regimes. The first is the usual symmetric label swap. The second is background-drawn noise, where corrupted positives are sampled from the background pool, as in the iNaturalist-BG and Places-BG constructions of Background Splitting.

To control clutter independently of label noise, we use the pixel-level foreground masks of RIVAL10, from A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes. We paste true foregrounds at controlled scale onto backgrounds drawn from other classes. This yields support sets where the foreground fraction and the corruption rate are set independently, and every example carries a ground-truth tag of clean, cluttered, or mislabeled.

That same paper constrains the design. It finds that robustly trained models lean more heavily on background. A denoiser that simply rewards stability could therefore amplify background shortcuts, so we measure where the maps land and do not stop at accuracy.

Baselines

The naive reference point is a plain prototype network on the same backbone and episodes, with no denoising. The file reports 57.07% for it at 40% label noise on miniImageNet (Rank-guided Diffusion for Noise Few-Shot Learning).

The strongest existing method is CRDProto, at 81.39% accuracy at 40% label noise on miniImageNet and about 4 seconds per task, from the same paper. CRDProto needs CLIP, so we run every method, ours included, on the CLIP image encoder for that comparison. This isolates the contribution of CRDProto's text prior from that of the backbone. On domains without meaningful class names we run CRDProto with its text term removed. Beating it means one of two outcomes:

  • matching it within noise on symmetric-noise miniImageNet while beating it on background-drawn noise and on clutter-composited RIVAL10;
  • or beating it outright wherever text priors are unavailable.

DETA is the closest method in mechanism. The file reports a 5.7% gain over a Swin Transformer at 30% label noise on Meta-Dataset (DETA). We run it with its published crop count and also at a crop budget matched to our number of forward and backward passes. To beat it we must win at matched compute and also label examples as clean, cluttered, or mislabeled more accurately than it does.

Peer-Collaboration Learning, from Learning to Learn from Corrupted Data for Few-Shot Learning, reports +5.27% over the prototype network at 40% noise on miniImageNet. It serves as the representative two-network filter.

Three ablations isolate the key ingredient:

  • a non-contrastive similarity map, $\langle A_{ij}, c_y^{(-x)}\rangle$, which tests whether the competitor-subtraction term matters;
  • an external-saliency mask in the style of SalNet, which tests whether the masks must come from the episode;
  • a two-way keep/drop variant with no crop option, which tests whether cropping is what rescues cluttered valid examples.

Every method shares backbones, episode seeds, noise seeds, and the same hyperparameter search budget on held-out validation classes.

Why not just…?

Why not just use a robust prototype, such as a medoid, a trimmed mean, or leave-one-out distance weighting? This is the most serious threat. For symmetric label swaps on uncluttered images, it may do as well as our method, and we will report that if it happens. The problem is that it judges whole embeddings. A real signal occupying a tenth of the image is as far from the class average as a mislabeled background event, so the robust estimator down-weights both. The composited RIVAL10 sets, with ground-truth example tags, settle the question. If leave-one-out weighting keeps cluttered valid examples as often as our three-way rule does, the saliency step is unnecessary.

Why not just use DETA's relevance-weighted random crops? DETA already separates clutter from wrong labels, but it does so by sampling crops without guidance. When signal is small, as in rare-event inputs, most random crops miss it. Our map points to the candidate region in a single backward pass. The matched-compute comparison, swept across foreground fraction, will show whether DETA's advantage holds once the signal shrinks.

Why not just use CRDProto, which already reports far higher numbers? On natural images with good class names it may remain best. Its own limitation is that CLIP text priors mislead it in specialized domains. It also treats each example globally, so it repairs an example without ever deciding whether the example was cluttered or mislabeled. We test on RIVAL10 and Background-Splitting-style sets with the text term removed, and with background-drawn noise. If CRDProto still wins there, our claim fails.

Why not just weight patches by similarity to the class prototype, skipping gradients? The derivation shows the gradient map is exactly a similarity map plus a competitor-subtraction term. The real question is whether that term matters. It should matter most under background-drawn noise, where background features resemble every prototype. The similarity-map ablation answers this directly. If the two perform equally, we will describe the method as contrastive patch scoring and drop the saliency framing.

Proposed Experiments

The first experiment replicates the setting where reported numbers exist: miniImageNet and tieredImageNet at 0–40% symmetric label noise, 5-way 1-shot and 5-shot. Its purpose is to check that we are competitive where CRDProto reports 81.39%.

The central experiment is the composited RIVAL10 grid. Foreground fraction and corruption rate vary independently, under both symmetric and background-drawn noise. We measure three things:

  • query accuracy;
  • area under the ROC curve (AUROC) for detecting mislabeled support examples;
  • three-way tagging accuracy against the ground-truth clean, cluttered, and mislabeled tags.

We also run a pointing-game check, asking whether the peak of each map falls inside the true foreground mask. This guards against the background-shortcut failure documented for RIVAL10.

For the rare-target regime, we build few-shot episodes from iNaturalist-BG with N=1 and N=10 rare classes, corrupted positives drawn from the background pool, and mean average precision (mAP) as the metric. Background Splitting's 52.9 mAP at N=1 was obtained with full supervision and is not a few-shot number. We cite it only as an upper reference, and there is no reported few-shot result on Lacuna for this setting. Meta-Dataset at 30% noise tests transfer across domains against DETA's reported gain.

A stretch goal tests the physics domain. If labeled germanium-detector waveforms comparable to the LEGEND Full Chain Test data in Machine learning-powered data cleaning for LEGEND are accessible, the same map computed over time samples should highlight pulse rise and trailing-edge regions for valid events, and should be diffuse for pileup and noise mislabeled as physics. This would test the method where text priors cannot exist. We found no few-shot result on Lacuna for this data.

Three outcomes would decisively count against the idea:

  • the similarity ablation matches the gradient map under background-drawn noise;
  • leave-one-out weighting tags cluttered examples as well as our method does;
  • the pointing-game hit rate falls as the corruption rate rises, meaning the maps follow the corruption rather than the signal.