Neighbor Evidence for Noisy Bag Labels in Long-Tailed Slide Classification

Slide-level cancer classifiers learn from whole-slide images that carry one diagnosis for hundreds of thousands of tissue patches. This setting is called multiple instance learning: each slide is a "bag" of unlabeled "instances" (patches), and only the bag has a label. Two things make real clinical datasets hard. Rare diagnoses have very few slides, and some slide labels are simply wrong. Methods built to clean noisy labels mostly treat a high training loss as a sign of a bad label. On rare classes this logic breaks, because a correctly labeled rare example also has high loss. In slides it breaks a third time: a rare-class slide whose tumor occupies a handful of patches has high loss early in training, before the model has found those patches. A filter that discards high-loss bags therefore removes the rarest and sparsest correct slides first. A model trained on what survives is then confidently wrong on the mislabeled slides it has memorized.

We propose a different test for whether a slide's label is believable. Do the few patches the model attends to look like patches that occur preferentially in other slides with the same label? We compute this in the frozen patch-feature space that slide classifiers already use, by checking how over-represented each label is among a patch's nearest neighbors, normalized by how common that label is. The score does not depend on the classifier's loss. Because the feature space is frozen, the classifier cannot memorize its way into a good score. Because the score is divided by class frequency, rare classes are not penalized for being rare. Because it looks only at attended patches, sparse evidence is not diluted by background tissue. The same score sets both the corrected training target and how confident that target is allowed to be. Noise correction and calibration, meaning the match between stated confidence and actual accuracy, then come from a single signal instead of two that work against each other.

Core Claim

In long-tailed slide classification with corrupted slide labels, a label-reliability score built from class-prior-normalized neighbor enrichment of attended patches in a frozen feature space can separate three cases that loss-based filters confound: mislabeled head-class bags, correctly labeled tail-class bags, and correctly labeled tail-class bags with very sparse evidence. Using this score as a per-bag soft target should improve tail-class F1 and per-class calibration error over loss-based and entropy-based alternatives under injected bag noise. It should also cost little on clean data.

The motivation comes from three bodies of work that each stop at a boundary. In the first, joint noise-and-imbalance methods are built for single images. CARE combines noisy labels with CLIP text and image experts through a Top-K consensus whose $K$ grows with class size. Unlocker learns the strength of logit adjustment in an outer optimization loop, using a balanced clean subset chosen by small loss. Both assume one image per label. Neither measures calibration.

In the second, long-tailed slide methods assume clean labels. DeCo-MIL documents the "nested dual long tail" of few rare slides and few evidence patches per slide. It shows that tail-class evidence is more concentrated, and it uses a pretrained judge for counterfactual patch interventions. MDE-MIL pairs balanced and unbalanced expert heads with distillation from the pathology vision-language model CONCH. Both assume pristine slide labels and report no calibration.

In the third, calibration methods for slides assume clean, roughly balanced data. UDLS smooths each slide's label in proportion to entropy under patch dropout. UFC-MIL smooths by patch-entropy statistics across resolutions. Both use entropy as the smoothing signal. Two papers constrain this design. To Smooth or Not? shows that uniform smoothing on noisy labels speeds memorization and erodes the clean/noisy loss gap. MIREL shows that attention weights reflect discriminative usefulness rather than certainty. We therefore use attention only to choose which patches to inspect, never as a confidence measure. Entropy also cannot be our noise signal, because a memorized mislabeled bag has low entropy while a clean tail bag has high entropy.

Research Program

The model is gated attention pooling in the form of Attention-based Deep Multiple Instance Learning, on precomputed, frozen patch embeddings. We use one attention branch per class so that every class has its own "top patches" in every bag.

For a patch $x$ from bag $b$, let $N_k(x)$ be its $k$ nearest neighbors in a pooled patch bank drawn from all other training bags. Let $\hat\pi_c$ be the fraction of the bank contributed by bags currently labeled $c$. The enrichment of class $c$ at $x$ is

$$E_c(x) = \frac{|\{z \in N_k(x) : \text{label}(z) = c\}| / k}{\hat\pi_c},$$

so background tissue that appears in every class scores near 1, and morphology specific to class $c$ scores well above 1. The bag's support for class $c$ is $S_b(c) = \mathrm{median}_{x \in T_b^c} \log E_c(x)$, where $T_b^c$ is the set of $m$ patches most attended by the class-$c$ branch.

The median over attended patches handles intra-slide sparsity, since only $m$ patches are needed. Dividing by $\hat\pi_c$ handles inter-slide rarity. Excluding the bag's own patches stops a mislabeled bag from vouching for itself. In a mislabeled bag, softmax attention must still put weight somewhere, and it lands on whatever patches best resemble the false class. These are typically background or other-class tissue, whose neighbors are not enriched for the false label. That is the mechanism we expect to separate it from a clean sparse tail bag.

The score enters training in three ways. First, the target for bag $b$ with observed label $y$ is

$$t_b = \lambda_b\, e_y + (1-\lambda_b)\, \mathrm{softmax}(S_b(\cdot)/\gamma),$$

where $e_y$ is the one-hot vector for $y$, $\gamma$ is a temperature, and $\lambda_b$ is a sigmoid of $S_b(y)$ relative to a per-class null. The null is the distribution of $S$ on randomly chosen patches of class-$y$ bags. This single target replaces both the keep-or-discard decision and the entropy-scaled smoothing of UDLS. Second, the class priors used for logit adjustment are re-estimated from the corrected targets rather than from the corrupted label counts. This is the issue Unlocker identifies, here handled without forming a clean subset. Third, every few epochs the patch bank's labels are replaced by the current corrected targets and the scores are recomputed, with a cap on how far a bag's label can move per round.

There is one warm-up phase on raw labels to make attention meaningful before scores are trusted. The neighbor search uses a subsampled bank of a few hundred patches per bag, which is cheap next to the multi-pass dropout of UDLS or the offline judge of DeCo-MIL.

We also run an analysis track to test the paper's motivating claim directly. On injected-noise splits we log attention entropy, top-attention mass, and bag-logit margin over training for clean and corrupted bags, separately for head and tail classes. This measures whether attention concentrates onto arbitrary patches in mislabeled bags and produces confident errors, a claim that is currently only argued in the literature.

Baselines

The naive reference point is gated attention pooling trained with plain cross-entropy on the noisy labels, with standard logit adjustment using the observed (corrupted) class counts. Every other method is measured against it, and it defines the scale of the problem.

The strongest existing long-tailed slide methods are DeCo-MIL and MDE-MIL. On Camelyon+-LT, MDE-MIL reports overall F1 of 63.02% and tail-class F1 of 28.57%, against 0.00% tail F1 for CLAM and DSMIL. DeCo-MIL reports an 18.82% tail-F1 gain over the best baseline there. Both numbers are on clean labels; no reported result found on Lacuna for any method on long-tailed slide benchmarks with noisy bag labels. We will rerun both on our noisy splits with the same frozen features, the authors' hyperparameters, and an equal tuning budget of the same number of validation configurations. Beating them means higher tail F1 at noise rates of 20% and above while matching them within run-to-run variance at 0% noise.

The strongest calibration competitor is UDLS. It reports an ECE drop from 0.202 to 0.077 for attention pooling on TCGA Lung (ECE, expected calibration error, is the average gap between stated confidence and accuracy over confidence bins). We run it with its two-stage procedure on the same splits. Beating it means lower per-class ECE on tail classes and on corrupted bags, not merely lower overall ECE.

For noise correction we transplant two single-image methods to bag inputs. The first is Unlocker-style small-loss selection with learned logit-adjustment strength, applied at the bag level. The second is a CARE-style consensus that uses CONCH zero-shot text scores and a fine-tuned slide head as experts, with class-adaptive $K$.

Three ablations isolate the key ingredients. The first computes the same enrichment in the trained model's own features rather than the frozen space, to test memorization immunity. The second removes the $\hat\pi_c$ normalization, to test class adaptivity. The third replaces the top-attended patches with all patches, to test the sparsity handling.

Why not just…?

Why not just use per-class small-loss thresholds and then temperature scaling? Ranking bags by loss within each class removes the rarity confound, and this is the most serious threat. It does not remove the sparsity confound. Within a tail class, clean slides with very little tumor still have the highest loss, so they are flagged first. Per-class thresholds also need a per-class noise rate that is unknown, and temperature scaling needs clean held-out bags. The experiment settles this directly: we stratify noise-detection AUROC by evidence concentration (the Gini measure from DeCo-MIL) within tail classes. If class-wise small loss matches our detection on the sparsest decile and matches our tail F1, the neighbor score is unnecessary and we will report that.

Why not just run CARE with a pathology vision-language model as the text expert? This keeps the class-adaptive consensus and swaps CLIP for CONCH, which MDE-MIL already shows carries tail-class signal. It is a real competitor, and we run it. Its weakness is that a slide-level zero-shot score averages over background tissue, so sparse lesions get weak text support. It also depends on prompt alignment for rare diagnoses. Our score needs no language at all. If CONCH-CARE wins on tail classes, the right next step is to feed its text score into $q_b$ rather than to defend the neighbor score.

Why not just apply UDLS or UFC-MIL and accept the noise? Both smooth according to entropy. To Smooth or Not? gives reason to expect that smoothing around wrong labels accelerates memorization, which would leave corrupted head bags with low entropy and little smoothing. Our analysis track measures exactly this: entropy on corrupted versus clean bags over training. If corrupted bags in fact keep high entropy under attention pooling, entropy smoothing is partially doing our job, and the gap to our method should shrink accordingly.

Proposed Experiments

We use Camelyon+-LT, PANDA-LT, and TissueNet-LT with the imbalance ratios of DeCo-MIL (30:1, 50:1, 10:1). We inject three kinds of bag noise at 0%, 20%, and 40%. The first is symmetric noise. The second is asymmetric noise between neighboring classes, for example adjacent grades in PANDA. The third is "evidence-free" noise, in which negative or low-grade slides are relabeled into tail classes, the case argued to cause attention-driven overconfidence.

The metrics are macro F1, tail-class F1, overall and per-class ECE reported separately for head and tail, the rate of confident errors (confidence above 0.9 on corrupted or misclassified bags), and AUROC for detecting corrupted bags, stratified by class and by evidence concentration. All results use five seeds with identical frozen features.

Three checks decide the claim. First, our score must assign clean sparse tail bags higher support than corrupted head bags, while small loss orders them the other way. Second, the frozen-space ablation must outperform the trained-feature ablation as noise rises, which tests whether immunity to memorization matters. Third, on PANDA-LT the method must not systematically flag lower-grade slides. Higher-grade prostate slides can also contain lower-grade patterns, which spreads low-grade enrichment across classes and could falsely mark clean low-grade bags as unsupported. If this occurs, the per-class null must be replaced with an ordinal-aware null, and we will report the failure rather than hide it.