Few-shot segmentation asks a model to outline a new structure, such as an organ never seen in training, given only one to five labeled example scans (the "support set"). The usual way to do this is to average the features inside the labeled region into a single "prototype" vector, then label each pixel of the new scan (the "query") by how similar it is to that vector. Recent work shows that the size of the region you average over matters a great deal. One global prototype per organ washes out sub-structure, while a grid of local prototypes keeps spatial detail but breaks when the query anatomy is shifted or deformed relative to the support. The right scale almost certainly differs by organ, by modality, and by position within an organ. Current methods fix it with a hand-chosen grid size or tuned propagation constants, or they learn a fusion module on base classes and hope it transfers.
We propose a different route. At test time, use the labeled support set itself to measure how well each scale of prototype would have worked, separately for each region of the image, and weight the predictions accordingly. No parameters are trained on the scarce target data. The support set already contains the evidence needed, through leave-one-out prediction across support scans, or across distant slices of a single support volume in the 1-shot case. The claim is that this support-calibrated selection beats both fixed-scale designs and learned global–local fusion modules on novel organs and modalities. We also expect the gains to concentrate exactly where fixed grids are known to fail: small, low-contrast, or displaced structures.
Core Claim
Global–local fusion under small-sample constraints is mainly a model-selection problem, not an architecture problem. Prior evidence shows the stakes are large. A Location-Sensitive Local Prototype Network builds masked-average prototypes inside overlapping spatial grid cells and matches query pixels only to the corresponding support cell. On VISCERAL it reports 70.3% Dice at a 1/8 grid, 30.3% with a single global prototype, and 60.2% with non-overlapping grids. A swing of 40 Dice points from one scale choice means the choice deserves a principled, per-case answer. The same paper also reports that the grid fails on small low-contrast targets such as the psoas, and that it assumes rigid support–query correspondence.
AGENet handles scale differently. It pairs spatially weighted global prototypes with an adaptive local grid that is denser near boundaries, using edge-aware geodesic distance fields. Its authors report that the propagation constants (iteration count $T$, speed scale $\lambda$) need delicate tuning, and small changes degrade results. We read both papers as motivating evidence: the best scale is real, large in effect, and not constant.
Our method removes the tuning by estimating, per region and per test episode, which prototype scale the support set itself supports.
Method
Start from any frozen feature extractor. We use two: the prototype-network encoder used by grid-prototype methods, and the frozen Swin-UNETR foundation model from Transductive few-shot adapters for medical image segmentation. From the support masks we build a small set of prototype "hypotheses" $s \in S$. These include one global prototype, grid prototypes at several cell sizes (1/4, 1/8, 1/16), and, for each grid size, a displacement-tolerant variant. In that variant a query pixel matches the best of the support cells within a radius $r$, rather than only the co-located cell. This tolerates the misalignment that breaks rigid grids. Each hypothesis produces a soft foreground map $p_s(x)$ over query voxels $x$.
The selector needs a reliability score for each hypothesis in each coarse region $c$ of the image. When $K \ge 2$, we predict each support scan from prototypes built on the other $K-1$ scans. We record the Dice of hypothesis $s$ inside region $c$, written $D_s(c)$, and average over held-out scans. When $K = 1$, we hold out slabs of the single support volume: we predict slices from prototypes built on slices at least a fixed gap away. We repeat this after random elastic deformations of the support, so the estimate reflects inter-patient misalignment and not only within-volume similarity. The fused prediction is
$$ p(x) = \sum_{s \in S} w_s(c(x))\, p_s(x), \qquad w_s(c) = \frac{\exp(\beta D_s(c))}{\sum_{s'} \exp(\beta D_{s'}(c))}, $$
where $c(x)$ is the region containing voxel $x$ and $\beta$ is a single temperature. We set $\beta$ once on base-class validation episodes and never retune it per dataset. The only free choices are the hypothesis set and $\beta$, both shared across all organs and modalities. Regions are coarse, roughly the 1/4 grid, so that each region's reliability estimate rests on enough voxels to be stable.
The design is deliberately low-capacity. Tartaglione et al. show that on small cohorts, extra capacity amplifies dataset-specific noise: ResNet-50 overfit where ResNet-18 did not. Learned fusion gates trained on a handful of scans risk the same failure. We therefore treat their result as a constraint that rules out heavy target-side fusion modules.
Research Program
The work has three parts. First, build the hypothesis bank and the leave-one-out estimator on top of two frozen backbones, and confirm that the estimator's ranking of scales correlates with the true query ranking. The estimator is only useful if what the support set "says" about scale matches what happens on the query. We measure this as the per-region Spearman correlation between $D_s(c)$ and true query Dice of each hypothesis.
Second, run the full method against fixed-scale, boundary-adaptive, recurrent, and learned-fusion competitors under one shared protocol.
Third, analyze where the selector moves weight, broken down by organ size, contrast, distance from the organ boundary, and support–query overlap. The aim is to test whether it recovers the known failure modes: shifting weight toward displacement-tolerant or global hypotheses for displaced or small organs, and toward fine grids for large organs with internal structure.
The method applies beyond segmentation. In few-shot classification, FeatWalk moves a global feature toward randomly sampled local patches, and its authors name sampling noise as an open issue. A support-calibrated choice of how far to walk is a natural follow-up, but we keep it out of scope so the medical study stays decisive.
Baselines
The naive reference point is a single global masked-average prototype per class, as in standard prototype networks, run on the same frozen features. Its expected weakness is documented: the global-prototype ablation in A Location-Sensitive Local Prototype Network falls to 30.3% Dice on VISCERAL. Our method must beat it by a wide margin or the whole premise fails.
The fixed-grid baseline is that paper's own method at its reported best 1/8 overlapping grid. It reports 70.3% mean Dice on VISCERAL 1-shot with auxiliary organ pre-training, and 66.7% without. We rerun it on our splits and encoder so the only difference from our method is the selection step.
The strongest existing method is AGENet. It already fuses global and boundary-weighted local prototypes and is evaluated across many modalities. It reports 81.67% mean Dice and 8.39 mm HD95 at 5-shot, and 79.56% Dice and 11.16 mm HD95 at 1-shot, averaged over its nine-dataset suite. To be fair, we run it with its published $T$ and $\lambda$ everywhere, matching our "tune once" rule. We report a second run in which $T$ and $\lambda$ are tuned per dataset on held-out base-class episodes, with the same number of validation episodes our $\beta$ received. Beating it means a higher mean Dice and a lower HD95 than the default-tuned AGENet on the shared suite, and at least parity with the per-dataset-tuned AGENet. That would show selection replaces tuning.
We also run Recurrent Mask Refinement (RP-Net), which contrasts foreground with neighboring background and refines the mask over 10 inference iterations. It reports 81.91% Dice on ABD-110, 72.48% on ABD-30, and 79.26% on ABD-MR at 1-shot. We report its latency next to ours, because single-pass speed is part of our claim.
For $K \in \{5, 10\}$ on TotalSegmentator and FLARE'22 we add the spatial adapters of Silva-Rodríguez et al. on the same frozen Swin-UNETR, with and without their transductive size prior. That paper reports per-organ gains rather than absolute FLARE'22 Dice in the material we have, so we will report only our reproduced numbers.
Three ablations isolate the key ingredient. The first is an equal-weight average over all hypotheses, which shows whether selection matters or diversity alone helps. The second is a single global choice of scale per dataset picked on base-class validation, which shows whether per-region, per-episode selection matters. The third is an oracle that picks each region's best hypothesis using the query labels, which bounds the possible gain. All methods use the same encoder, splits, preprocessing, and number of validation episodes for any tuned constant.
Why not just…?
Why not just average all scales with equal weight? Ensembling is the cheapest form of fusion and often works. This is the most direct threat, and the equal-weight ablation tests it. If the equal-weight ensemble comes within 0.5 Dice of our method on average and on the small-organ subset, we drop the claim that selection matters and report the ensemble as the practical recommendation. We expect selection to win mainly where one hypothesis is badly wrong, for example the global prototype on heterogeneous organs. An unweighted average lets that hypothesis drag the result down, and the 30.3% vs 70.3% gap on VISCERAL suggests this happens.
Why not just meta-learn a gating network on base classes? A learned gate, like the fusion modules of AGENet or QGNet, sees many base-class episodes and could learn when local detail helps. QGNet shows that jointly training global and local contrastive objectives beats either alone, so learned global–local interaction does carry signal. The risk is transfer: a gate trained on abdominal CT organs encodes those organs' sizes and shapes. We test this directly. We train a small gate on base CT classes, using the same hypothesis bank and the same per-hypothesis predictions as input, and evaluate it on held-out organs and on the MRI and non-CT datasets of the AGENet suite. If it matches our selector on in-domain organs but loses on the cross-modality datasets, that supports using test-time evidence. If it wins everywhere, the support-calibrated estimate is not needed and we will say so.
Why not just register support to query and then use a fixed grid? Misalignment is the stated failure of rigid grids, and image registration is the classical fix. RP-Net already relies on an initial affine alignment. This is a real competitor, so we include it two ways: as a standalone baseline (affine plus deformable registration, then the 1/8 grid) and as one more hypothesis inside our bank. If registration plus grid matches the full method, our contribution narrows to "choosing among scales after alignment." That is still useful, but much smaller, and the analysis will show it by revealing whether the selector puts nearly all its weight on the registered hypothesis.
Why not just fine-tune adapters on the support scans? When $K$ is 5 or 10, training a small adapter is feasible. Silva-Rodríguez et al. report that spatial adapters beat full fine-tuning in the 1-shot regime. Adapters are orthogonal to our method, because the selector can run on adapter features. We test both separately and combined. At $K=1$, we expect training-free selection to be competitive because no adapter can be validated on held-out data.
Proposed Experiments
We evaluate 1-shot and 5-shot on VISCERAL using the 65/20 split of A Location-Sensitive Local Prototype Network; on ABD-110, ABD-30 and ABD-MR as in RP-Net; and on the multi-modality suite used by AGENet (abdominal CT and MR, ACDC, PH2, CVC-300, CHASE-DB1, Drishti-GS, DCA1). We add 1-, 5- and 10-shot runs on TotalSegmentator and FLARE'22 with the frozen Swin-UNETR. Metrics are Dice, HD95, per-organ Dice binned by organ volume and contrast, and wall-clock inference time per volume. The leave-one-out step multiplies prototype computation by roughly $K$, or by the number of held-out slabs at $K=1$, and we report this cost honestly.
Three outcomes would settle the question. First, the support-estimated reliabilities must predict query reliabilities: a per-region rank correlation clearly above zero across held-out organs. Without this, the mechanism is unsupported regardless of headline Dice. Second, the full method must beat the fixed 1/8 grid and the default-tuned AGENet on the shared suite, and match or beat per-dataset-tuned AGENet with no per-dataset tuning. The largest gains should appear on small or low-contrast organs and on cases with low support–query overlap. Third, the gap to the equal-weight ensemble and to the meta-learned gate on cross-modality datasets decides whether test-time selection is the right lever. We commit in advance to reporting a negative result if either of those simpler alternatives closes the gap.
References
- A Location-Sensitive Local Prototype Network For Few-Shot Medical Image Segmentation
- AGENet: Adaptive Edge-aware Geodesic Distance Learning for Few-Shot Medical Image Segmentation
- Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentation
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- FeatWalk: Enhancing Few-Shot Classification through Local View Leveraging
- Recurrent Mask Refinement for Few-Shot Medical Image Segmentation
- Few-Shot Segmentation with Global and Local Contrastive Learning