When a pathology model has only a handful of labeled whole-slide images, the gigapixel scans cut into thousands of small patches, recent evidence says the smartest aggregator is almost no aggregator. Average the frozen foundation-model embeddings of all patches, then train a two-layer network on top. This beats attention-based pooling and is more stable across hospitals. That simplicity breaks exactly where it matters clinically. When the diagnostic evidence sits in a few dozen patches out of thousands, as with a small focal lesion or a rare biomarker, averaging drowns the signal. The usual fix is a learned attention module that picks the important patches, but it brings back the extra capacity that overfits a tiny cohort. We propose a third option: an aggregator with no learned parameters. It scores each patch by how much more it resembles patches from one class than from the others, using nearest-neighbor distances in the frozen embedding space. It then averages only the top fraction of patches. That fraction is the single capacity knob, and it is chosen by leave-one-slide-out accuracy, which is cheap to compute exactly, so no held-out validation set is needed.
In two sentences: sparse evidence can be located without learning anything, by asking which patches are unusually close to one class's patches and far from the others'. The only decision left, how many patches to trust, can be made on the training slides themselves. That gives one method that behaves like averaging on diffuse tasks and like max-pooling on needle-in-a-haystack tasks, with no validation set and no attention weights to overfit.
Core Claim
Our claim is that the complexity trade-off in few-shot slide classification is mostly about where the pooling looks, not about how many parameters the head has. A zero-parameter, density-ratio patch selector whose sparsity is set by leave-one-slide-out should do three things. It should match the simple averaging baseline on diffuse tasks, beat it on sparse-evidence tasks, and match or beat attention-based pooling at 1, 5 and 10 slides per class, with lower variance across seeds.
For a patch embedding $x$ from a frozen encoder of dimension $D$, and class $c$, let $r_k(x; c)$ be the distance to the $k$-th nearest patch among training slides of class $c$, and $r_k(x; \neg c)$ the same among all other classes. The patch's own slide is always excluded from both sets. The patch score is the standard $k$-nearest-neighbor log density ratio,
$$s_c(x) = D \,\log \frac{r_k(x;\neg c)}{r_k(x; c)},$$
which follows from the $k$-NN density estimate scaling as $r_k^{-D}$. The slide-level score for class $c$ is the mean of $s_c$ over the top fraction $q$ of that slide's patches. The prediction is the class with the highest score. Setting $q=1$ gives a density-weighted analogue of averaging, and small $q$ approaches max-pooling. Because every training slide is scored with its own patches excluded, leave-one-slide-out accuracy for every candidate $q$ comes from the same neighbor queries. $q$ is then picked from a grid by that accuracy. The top-$q$ patches also serve directly as the heatmap a pathologist would inspect.
Relation to Prior Work
The simple-baseline results motivate the proposal. Can We Simplify Slide-level Fine-tuning of Pathology Foundation Models? shows that averaging frozen UNI/CONCH/Prov-GigaPath patch features and training a two-layer network (SiMLP) beats attention-based multiple-instance pooling (ABMIL) and is more stable across hospital cohorts. The same paper concedes that it fails on sparse-evidence tasks and gives no heatmaps. Our method keeps the frozen features and the low head capacity but replaces uniform averaging with class-conditioned selection. Unravelling Small Sample Size Problems in the Deep Learning World motivates the other side: global averaging washes out sparse signals, and learned sliding-window attention pooling helps on CIFAR and TinyImageNet. Its fix adds learnable parameters, which is exactly what SiMLP found overfits in pathology. We aim for its benefit with none of its parameters.
The method comes from Simpler Non-Parametric Methods Provide as Good or Better Results to Multiple-Instance Learning. That work shows $k$-NN and Parzen-window scoring beats parametric multiple-instance methods and tolerates label noise, but on hand-crafted features and with gradient-ascent prototype search that does not scale to slide cohorts. We move the idea onto foundation embeddings, drop the prototype search, and add the leave-one-out sparsity selection. The validation-free principle comes from Few-Shot, Now for Real, whose SS-Text+ uses a closed-form solver and class-adaptive regularization to avoid spending scarce samples on validation. SS-Text+ is linear and image-level. Our method is non-linear through the neighbor distances and bag-level.
Two constraints shape the design. NCT-CRC-HE: Not All Histopathological Datasets are Equally Useful shows that color-histogram features alone reach 82.2% on that benchmark because of JPEG and color-calibration artifacts. A nearest-neighbor selector could just as easily lock onto stain or site signatures. We therefore treat shortcut susceptibility as a primary evaluation axis, not an afterthought. Towards Bridging Sample Complexity and Model Capacity ties required capacity to cluster structure with a fixed distance threshold. We use its "dominating points" count only as a diagnostic, to test whether the selected $q$ tracks the cluster structure of the class-discriminative patches. We do not claim a bound.
Research Program
The build is small on purpose. Patches are embedded once with frozen UNI and CONCH. Each slide is subsampled to a fixed patch budget, around two thousand, so that approximate nearest-neighbor search over the training pool stays cheap even with hundreds of slides. The first analysis maps the regime structure: how the selected $q$ varies with slides per class and with the true fraction of evidence-bearing patches. We expect $q$ near 1 on diffuse tasks such as pan-cancer typing and small $q$ on sparse tasks. If that holds, the method replaces a static architecture choice (averaging or attention) with a data-driven one. A second variant feeds the top-$q$ pooled embeddings into SiMLP's two-layer head. This tests whether any trained capacity helps once selection is fixed. We expect it to help only at the largest cohort sizes.
Baselines
The naive reference is SiMLP itself, meaning mean pooling plus a two-layer network as described in Can We Simplify Slide-level Fine-tuning of Pathology Foundation Models?. It is also the strongest existing method for the diffuse regime. Its reported margin is +3.52% balanced accuracy over ABMIL on TCGA pan-cancer. The file gives no absolute SiMLP balanced accuracy, so we will reproduce it. On diffuse tasks, beating it means staying within its seed-to-seed standard deviation. On sparse tasks, beating it means a gain larger than two standard deviations. ABMIL is the strongest competitor on sparse evidence because it is the standard learned selector. We run it on the same frozen features, with the same slides and seeds and an equal tuning budget. Since our method uses no validation set, ABMIL gets two settings: a fixed default recipe, and an oracle setting tuned on a held-out split drawn from outside the training budget. The oracle setting is an upper bound on what it could achieve. Beating fixed-default ABMIL and staying within its standard deviation under oracle tuning would count as success. A non-parametric reference, $k$-NN with patch labels inherited from the slide and majority voting, isolates whether the density ratio and top-$q$ pooling add anything beyond plain neighbor voting. For the classic $k$-NN multiple-instance approach, the file reports 81.3% on diabetic retinopathy from Simpler Non-Parametric Methods. We found no reported result on Lacuna for it on foundation-model slide features.
Three ablations isolate the key ingredient. The first fixes $q$ at 1, which tests whether selection matters. The second fixes $q$ at a single global value, which tests whether leave-one-out selection matters. The third replaces the density-ratio score with a class-agnostic top-$q$ selection such as embedding norm or distance from the slide mean. This tests whether the class-conditioned score is doing the work.
Why not just…?
Why not just use top-k or generalized-mean pooling with a tuned exponent? This is the most serious threat. It also interpolates between averaging and max-pooling with one scalar. The difference is that it pools by feature magnitude, not by class evidence, and tuning its scalar needs a validation set. The class-agnostic ablation above, given the same leave-one-out selection of its scalar, settles the question. If it matches our method, the contribution shrinks to "leave-one-out selection of pooling sparsity," and we will report it that way.
Why not just regularize attention pooling harder? Stronger dropout or weight decay on ABMIL might close the variance gap. The SiMLP study reports that attention pooling overfits and varies across seeds precisely in the 1–10 slide regime, and strong regularization pushes attention toward uniform weights, which is averaging again. The oracle-tuned ABMIL baseline gives attention its best chance. If it still loses on variance at 1 and 5 slides per class, the case is made.
Why not just train a patch-level classifier with slide labels and take the max? With one to ten slides per class, inherited patch labels are overwhelmingly wrong on sparse tasks, since most patches in a positive slide are negative. A trained patch classifier fits that noise. The inherited-label $k$-NN baseline is the non-parametric version of this idea and tests it directly.
Why not just generate more slides? Limitations of Synthetic Data Generation in Specialized Data-Scarce Domains found that no generative method consistently beat AugMix, and that synthetic images were easier than real ones by 11–45% recall. That is the opposite of what a sparse-evidence problem needs.
Proposed Experiments
We use the SiMLP benchmark suite: TCGA pan-cancer classification, NSCLC subtyping across TCGA, CPTAC and an independent cohort, and its biomarker-prediction tasks. Each runs at 1, 5, 10 slides per class and full data, over at least ten seeds. We report balanced accuracy, ROC AUC and seed standard deviation.
Because the file names no standard sparse-lesion slide benchmark, we build a planted-sparsity benchmark. Tumor-labeled patches from TCGA slides are inserted into normal-tissue slides at controlled fractions from 50% down to 0.5%. Ground-truth locations are then known, so we can measure both accuracy and how well the selected patches overlap the planted ones. The decisive check is the crossover curve: accuracy against planted fraction for SiMLP, ABMIL and our method. We predict SiMLP collapses as the fraction falls, ABMIL degrades at low slide counts, and our method holds on both axes, with the selected $q$ falling alongside the planted fraction.
A second, harder check targets shortcuts. We rebuild the planted benchmark from NCT-CRC-HE patches, then re-run the evaluation after JPEG recompression and train on one site while testing on another for NSCLC. If the density-ratio selector concentrates on patches whose selection flips under recompression, it is reading artifacts, as NCT-CRC-HE warns. That failure would be reported as a limit of the method. Finally, we compute the dominating-point count of Towards Bridging Sample Complexity and Model Capacity on the top-scoring patches and test whether it correlates with the selected $q$ across tasks. A positive correlation would link the empirical sparsity knob to a measurable property of the data.
References
- Can We Simplify Slide-level Fine-tuning of Pathology Foundation Models?
- Unravelling Small Sample Size Problems in the Deep Learning World
- Simpler Non-Parametric Methods Provide as Good or Better Results to Multiple-Instance Learning
- Few-Shot, Now for Real: Medical VLMs Adaptation Without Balanced Sets or Validation
- NCT-CRC-HE: Not All Histopathological Datasets are Equally Useful
- Towards Bridging Sample Complexity and Model Capacity
- Limitations of Synthetic Data Generation in Specialized Data-Scarce Domains