Pathology slides are gigapixel images, far too large for a GPU, so they are cut into thousands of small tiles and classified using only one diagnosis per slide. This setup is called multiple instance learning (MIL): the slide is a "bag" and the tiles are unlabeled "instances." The standard model scores every tile, turns the scores into attention weights with a softmax over the whole bag, and pools. A second model, an instance classifier, is often trained on those attention weights as soft per-tile pseudo-labels, so that it can point to where the tumor is. This proposal starts from one observation. A softmax taken over ten thousand tiles is the same operation that erases "dark knowledge" in ordinary distillation, where the teacher's graded preferences among non-winning options get squashed to near zero. When the teacher is confident, the tiles it does not rank first all collapse toward zero weight. Their relative ordering, which is where faint tumor margins and small metastases live, disappears before the instance classifier ever sees it.
The core claim is that the well-documented "model laziness" of attention MIL is dark-knowledge degradation along the instance axis rather than the class axis. If so, a counterintuitive prediction follows. Stronger, domain-specialized pathology foundation models produce more separable tile features. More separable features produce peakier attention. Peakier attention should transfer less localization information to the instance classifier, which mirrors the capacity-mismatch result that bigger teachers can make worse students. We propose to test this prediction directly across a ladder of feature extractors. We then test whether a calibration of attention targets that preserves the structure of the tail lets the instance classifier exceed its own bag-level teacher at localization.
Core Claim and Grounding
In class-level distillation, Li et al. (ATS) showed that useful information lives in the variance of the non-target logits. Softmax exponentiation of large logit gaps drives that variance to numerical insignificance. Their fix applies separate temperatures to the target class and to the rest. Dao et al. give a complementary account: overfit teachers collapse toward one-hot outputs and reduce the student to plain supervised learning. Both results motivate this proposal, but both treat the softmax as running over classes for one i.i.d. image. In MIL the softmax runs over instances within one bag. The "target" is a handful of obvious tumor tiles, and the "non-target tail" is everything else.
WENO is the closest existing work, and the proposal takes its setup as the starting point. WENO trains an instance student on normalized attention as soft pseudo-labels, and it counters laziness by masking tiles the student is already confident about (hard positive mining). WENO treats peaky attention as a coverage problem to be solved by masking. We treat it as a target-calibration problem, to be measured and corrected in the pseudo-labels themselves. WENO also used non-pathology features, so it could not observe how the effect scales with feature strength. LadderMIL supplies a theoretical constraint we build on: slide-level convergence does not guarantee instance-level learnability. Its remedy is hard top-k pseudo-labels on a fixed 5%→20% schedule, which discards the tail ordering entirely. Moniri et al. provide the reason a student could beat its teacher here. A student with strong domain-aligned priors can see through an over-confident teacher's errors, and frozen pathology foundation model features are exactly such a prior. Domain specialization therefore plays two roles in this proposal. It is the suspected cause of the degradation, and it is the resource that should let the student break the teacher ceiling.
Research Program
The first part is a diagnostic study. For a bag with tile scores $s_i$ and attention $a_i = \exp(s_i)/\sum_j \exp(s_j)$, we define the anchor set $A$ as the top-$k$ tiles and the tail $T$ as the rest. We measure three things. The first is the effective number of attended tiles, $\exp(H(a))$, where $H$ is the entropy of the attention. The second is the within-tail rank correlation between $s_i$ and the annotated tumor fraction of each tile, using CAMELYON16's pixel-level masks. The third is the numerical mass of the tail attention. The hypothesis has two parts. As the extractor moves from ImageNet ResNet50 through progressively stronger pathology foundation models (UNI, Virchow2, H-Optimus-0, Prov-GigaPath, the models HistoMILKD distills), the raw scores $s_i$ should still rank tail tiles reasonably. However, the attention mass on the tail should collapse, and the instance classifier distilled from $a_i$ should localize worse than its own scores would allow.
The second part is a fix that operates on the pseudo-label targets, not on the aggregator. We standardize tail scores within each bag, in the spirit of LumiNet, but restricted to the tail so that the few anchor tiles do not dominate the statistics:
$$y_i = \sigma\!\left(\beta \,\frac{s_i - \mu_T}{\sigma_T}\right) \text{ for } i \in T, \qquad y_i = 1 \text{ for } i \in A \text{ in positive bags},$$
Here $\mu_T$ and $\sigma_T$ are the mean and standard deviation of scores over the tail of that bag, $\sigma$ is the logistic function, and $\beta$ is a single sharpness parameter. This is the instance-axis analogue of asymmetric temperatures: anchors are clipped rather than allowed to dominate, and the tail's relative structure is expanded rather than exponentiated away. LumiNet's known weakness is sensitivity to small batches. That weakness matters little here, because a bag supplies thousands of instances. Its other weakness, fragility under class imbalance (Gao et al.), is the reason anchors are excluded from the statistics: tumor burden below 5% is extreme within-bag imbalance. The bag aggregator is trained as usual. The instance student, a small head on the same frozen features, is trained with binary cross-entropy against $y_i$, and negative bags use target 0.
The third part tests the ceiling. Using CAMELYON16 annotations, we ask whether the instance student's localization exceeds that of the teacher's own attention map. If it does, the student has broken the teacher-dependent ceiling in the sense of Moniri et al. We also report whether the effect grows with extractor strength, as their theory predicts for students with better priors.
Baselines
The naive reference point is standard attention MIL (ABMIL) on frozen features, with the attention map used directly as the localization output. On CAMELYON16, WENO reports an instance-level AUC of 0.8480 for ABMIL. The strongest existing competitor on this question is WENO itself, which reports 0.9271 instance-level AUC on CAMELYON16 with bi-directional distillation and hard positive mining. It is the strongest because it attacks the same failure with the same teacher–student structure. Both published numbers use non-pathology features, and we found no reported instance-level result on Lacuna for any of these methods with pathology foundation model features. We will therefore rerun all methods ourselves on each extractor. For slide-level context we also rerun DSMIL, which reports 89.9% accuracy and a localization FROC of 0.437 on CAMELYON16, and TransMIL, which reports a test AUC of 93.09% on CAMELYON16, 96.03% on TCGA-NSCLC, and 98.82% on TCGA-RCC, using its public codebase. We will also rerun LadderMIL's top-k scheduled pseudo-labels.
Fairness means identical frozen features per extractor, identical official splits, the same aggregator architecture, five seeds, and an equal hyperparameter budget of twenty configurations per method, chosen on a validation fold. Our single $\beta$ and $k$ receive no more tuning than WENO's masking threshold or a temperature sweep. Beating WENO means a higher instance AUC and FROC on CAMELYON16 averaged over seeds, with non-overlapping 95% bootstrap intervals, on at least the two strongest extractors, and no loss in slide-level AUC.
Three ablations isolate the key ingredient. A global-temperature baseline replaces our target with $a_i$ computed at a single swept temperature. A rank-transform baseline uses per-bag score quantiles, which is also immune to peakiness. The decisive ablation is a tail-shuffle control. It keeps the exact marginal distribution of $y_i$ over the tail but randomly permutes which tail tile receives which target. If localization under the shuffle matches the unshuffled targets, the ordering of the tail carries no information, and the central claim is false.
Why not just…?
Why not just tune a single softmax temperature on the attention? This is the most serious threat. ATS showed that one temperature fails for class logits, because raising it flattens non-target differences while lowering it leaves the target dominant. Nobody has shown this holds along the instance axis, where the tail is thousands long rather than dozens. The global-temperature and rank-transform ablations settle it. If either matches tail standardization within noise, we will report that the contribution is the diagnosis and the scaling finding, and that the simplest calibration suffices. That would still be a useful result, but a smaller one.
Why not just use WENO's hard positive masking? Masking changes which tiles the teacher attends to, but each round still produces peaky targets over whatever tiles remain, so the tail structure is lost at every round. Masking also requires a grid-searched threshold. We test the two ideas as complementary. The masking-plus-calibration combination should beat masking alone if calibration adds information, and calibration alone versus masking alone tells us which lever matters more for each extractor.
Why not just use a stronger foundation model and rely on better features? The diagnostic study exists to answer exactly this question. Our hypothesis is that stronger features raise slide-level accuracy while flattening the attention tail, so that attention-based localization plateaus or declines. If instead instance AUC from plain ABMIL rises steadily with extractor strength, and the tail statistics do not collapse, the premise is wrong and the idea should be dropped. A related alternative is to regularize attention entropy during aggregator training, the analogue of label smoothing. Müller et al. and Chandrasegaran et al. found that label smoothing flattens non-target structure toward uniform rather than preserving it. We include an entropy-regularized aggregator as a baseline to check whether that constraint transfers.
Proposed Experiments and Decisive Checks
The main experiments use CAMELYON16, because its tumor annotations allow instance AUC and FROC. They cross five extractors (ImageNet ResNet50 plus four pathology foundation models) with every method above. TCGA-NSCLC and TCGA-RCC test that slide-level balanced accuracy and AUC are not sacrificed. On these datasets, we qualitatively review the heatmaps against the teacher attention maps. Camelyon+-LT tests the regime where rare classes hide in the tail. There, MDE-MIL reports an overall F1 of 63.02% and a tail-class F1 of 28.57%, while DSMIL and CLAM score 0.00% on tail classes. We ask whether tail-preserving targets improve rare-class recall of the instance student without any rebalancing branch.
Three results decide the project. The first is a negative correlation, across extractors, between extractor strength and the effective number of attended tiles, together with a widening gap between the localization of raw scores and that of attention-distilled students. The second is that the unshuffled tail targets beat the tail-shuffle control, which shows that tail ordering is the transferred knowledge. The third is that the instance student exceeds the teacher's own attention map at localization, and does so by more on stronger extractors. If the first check fails, the phenomenon does not exist in this regime. If the second fails, the mechanism is wrong even if the numbers improve.
References
- Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again
- Knowledge Distillation as Semiparametric Inference
- Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classification
- Multiple Instance Learning with Coarse-to-Fine Self-Distillation
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective
- HistoMILKD: A Multiple Instance Learning based Multi-Teacher Knowledge Distillation Framework for Whole Slide Image Classification
- LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration
- Maintaining Fairness in Logit-based Knowledge Distillation for Class-Incremental Learning
- Dual-Stream Multiple Instance Learning Network for Whole Slide Image Classification With Self-Supervised Contrastive Learning
- TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification
- When does label smoothing help?
- Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?
- Multimodal Distillation-Driven Ensemble Learning for Long-Tailed Histopathology Whole Slide Images Analysis