Models that diagnose a whole-slide pathology image cut the slide into tens of thousands of small tiles and learn from a single label per slide, such as "contains metastasis." This setup is called multiple instance learning (MIL). These models now classify slides almost perfectly, but the per-tile scores they produce, which pathologists read as a heatmap of where the cancer is, are poor. On CAMELYON16, standard attention MIL reaches a slide AUC of 0.895 but a mean localization Dice of only 0.215 (Attention-based Multiple Instance Learning with Mixed Supervision on the Camelyon16 Dataset). Its heatmaps scatter false positives across benign tissue and blur across tumor margins. We propose to treat that heatmap as the unknown signal in a classical image-restoration problem. The raw tile scores are a noisy observation. The cleaned map is recovered by minimizing two penalties. The first is a local penalty, total variation on the physical tile grid: the sum of absolute score differences between neighboring tiles, which allows sharp boundaries but punishes speckle. The second is a nonlocal penalty that pulls together the scores of tiles that look alike anywhere on the slide, in the spirit of non-local means.
The solver runs a fixed number of primal-dual iterations, unrolled inside the network and trained end to end from slide labels alone. It operates on one scalar per tile, not on high-dimensional features, and every step is a sparse graph operation. Its cost is therefore linear in the number of tiles, even at 50,000+ tiles at 40× magnification. The central claim is that coupling a sharp-edged local prior with a morphology-driven nonlocal prior, inside the forward pass, can recover much of the localization gap that otherwise needs patch annotations, without lowering slide-level accuracy.
Core Claim and What Is New
Classical hybrid regularization shows that each prior fails alone in a predictable way. Low-rank or nonlocal priors alone over-smooth unique structure. Total variation alone produces blocky "staircase" artifacts and cannot exploit long-range repetition. Combining them fixes both, as in tensor-rank-plus-TV completion solved by primal-dual splitting (Simultaneous Visual Data Completion and Denoising Based on Tensor Rank and Total Variation Minimization and Its Primal-Dual Splitting Algorithm), low-rank-plus-TV super-resolution (Low-Rank Total Variation for Image Super-Resolution), and TV-plus-nonlocal compressive sensing recovery (Improved Total Variation based Image Compressive Sensing Recovery by Nonlocal Regularization). These works supply our method.
The MIL work that touches this territory uses only one half of the pair, and uses it on features rather than on the output map. Exploring Low-Rank Property in Multiple Instance Learning for Whole Slide Image Classification imposes a global low-rank structure on tile embeddings and routes attention through a latent bottleneck. It has no spatial or edge term, and it gives up direct per-tile attention. Exploiting Label-Independent Regularization from Spatial Patterns for Whole Slide Image Analysis adds a spatial graph attention network with masked feature reconstruction. That is local only, and it uses the quadratic, low-pass kind of graph smoothing that Total Variation Graph Neural Networks shows blurs cluster boundaries. TVGNN itself replaces quadratic smoothing with an ℓ1 graph total variation. However, it works on high-dimensional node features, needs an approximation because exact per-dimension TV is costly, and was tested only on small citation and molecule graphs.
Our proposal differs from all three in four ways. It applies the hybrid penalty to the scalar score field, which makes exact ℓ1 TV cheap. It keeps per-tile scores fully interpretable. It couples local and nonlocal terms in a single solver. And it is evaluated mainly on localization, where slide-level methods have stalled, rather than on slide AUC, which is near saturation.
The Refinement Layer
Let $a \in \mathbb{R}^n$ be the raw attention logits of a standard gated-attention MIL head over the $n$ tiles of a slide. Let $E_{sp}$ be the edges of the 8-neighbor tile grid, and let $E_{nl}$ be the edges of a $k$-nearest-neighbor graph built in frozen feature space. Each nonlocal edge has a weight $w_{ij}$ from a non-local-means-style kernel on feature distance. The refined scores $s$ solve
$$\min_{s}\; \beta \sum_{(i,j)\in E_{sp}} |s_i - s_j| \;+\; \alpha \sum_{(i,j)\in E_{nl}} w_{ij}\,(s_i - s_j)^2 \quad \text{subject to} \quad \|s - a\|_2^2 \le \delta n .$$
Here $\beta$ and $\alpha$ weight the local and nonlocal terms. $\delta$ bounds how far, on average, refinement may move the raw scores.
Following LRTV-PDS, we replace a free data-fidelity weight with this ball constraint. The constraint is convex and needs no label, so training and test use the same forward pass. It also removes one of the hand-tuned balance parameters that the file flags as an open problem for hybrid models.
The solver is primal-dual splitting. It needs only the proximal map of the absolute value (soft thresholding), projection onto a ball, and sparse multiplications by the two graph difference operators. It requires no singular value decomposition and no matrix inversion. We unroll $K\approx 10$–$20$ iterations, learn $\alpha$, $\beta$, $\delta$ (kept positive) by backpropagation, and feed the refined scores into the usual softmax attention pooling and slide classifier.
The cost per slide is $O(K(|E_{sp}|+|E_{nl}|)) = O(Knk)$. Backpropagation stores $K$ length-$n$ vectors, which is trivial next to the tile features. The one-time approximate $k$-nearest-neighbor search is the only global operation.
The design choice is deliberate. The local term is ℓ1, so tumor boundaries on the physical slide stay sharp. The nonlocal term is quadratic over morphology, so a distant tile that looks like a confident tumor tile is pulled toward a high score. This suppresses isolated benign false positives, and it also lifts small metastatic foci that look like confident ones. We ablate the opposite pairing rather than assume this one is right.
Research Program
The program has three parts.
The first builds the layer on top of a fixed pipeline: CLAM-style tiling, one frozen feature extractor shared by all methods, and gated attention. We verify that the unrolled solver converges on real slides at 20× and 40×, and we measure wall-clock time and memory against tile count.
The second asks whether the refinement must live inside training. We compare three versions. The full in-loop layer is trained end to end. A loss-only variant adds the same two penalties to the training loss but uses raw scores at inference. A post-hoc variant runs the same solver only at test time on a model trained without it.
The third asks what each prior contributes and how they interact. We compare TV-only, nonlocal-only, both terms quadratic, both terms ℓ1, and swapped norms. We also test sensitivity to $k$ and to the nonlocal kernel bandwidth, and we check whether the learned $\alpha/\beta$ ratio is stable across datasets. A stable ratio would partly address the balancing problem noted for LRTV-PDS.
Baselines
The naive reference point is CLAM, the gated-attention MIL model with instance clustering (Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images). This is also our backbone, so the comparison isolates the added layer. The mixed-supervision study reports this model at slide AUC 0.895 and mean Dice 0.215 on CAMELYON16. We rerun it on our features and splits rather than copying that number.
The strongest slide-level competitors are the following. ILRA-MIL reports 96.49% AUC on CAMELYON16 and is the closest existing low-rank method. Snuffy (Snuffy: Efficient Whole Slide Image Classifier) reports AUC 0.987 and, importantly for us, FROC 0.675 for region detection on CAMELYON16. That makes it the strongest reported localization result in the file and the number our method must approach or exceed. TransMIL (TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification) reports 93.09% AUC and is the representative of linear attention with convolutional position encoding. SRMIL reports 91.2% accuracy and is the representative spatial-graph regularizer.
For fairness, every method is retrained from public code where it exists (the TransMIL repository, CLAM). All use the same frozen feature extractor, the official 270/129 CAMELYON16 split, the same number of hyperparameter trials (for example 20 random-search trials per method on a held-out training fold), and three seeds. Our method's internal parameters are learned and count against the same budget. Snuffy's reported numbers rely on its own fine-tuned ViT, so we report both its published figure and a run on shared features. Localization heatmaps are read directly from each method's per-tile scores. ILRA-MIL has no direct per-tile attention, so we use its first-stage attention to the latent matrix and say so.
The upper reference is CLAM trained with 10% of tile annotations, reported at Dice 0.350 and AUC 0.924. Beating our method means, with no tile labels, matching or exceeding the rerun annotation-free baselines on slide AUC (within one seed standard deviation of the best) while clearly exceeding them on Dice and FROC. The headline target is reaching or passing the 10%-annotation Dice. The ablation that isolates the key ingredient is the swap of the ℓ1 local term for a quadratic one with everything else fixed. If that swap costs nothing, the edge-preservation argument fails.
Why not just…?
Why not just add a TV penalty on the attention map to the training loss? This is the most serious threat, because it is simpler and might give most of the gain. Our loss-only ablation is exactly this alternative with identical penalties and weights. If it matches the in-loop layer on Dice and FROC, we will report that the prior, not the unrolled solver, is the contribution, and we will recommend the cheaper form. We expect the in-loop version to win for one reason: a training penalty shapes average behavior, but it does not guarantee that a given test slide's map is piecewise-smooth. The solver enforces this on every slide.
Why not just smooth the heatmap after the fact, with a Gaussian filter or a conditional random field? Post-hoc filtering uses only spatial proximity. It cannot raise a small isolated metastasis that resembles confident tumor elsewhere on the slide, which is the job of the nonlocal term. Gaussian smoothing also blurs margins, which is the low-pass failure documented by TVGNN. Our post-hoc variant runs the full hybrid solver at test time only, and we add a plain Gaussian-smoothing control, so this question is settled directly.
Why not just use TransMIL or a spatial GNN, which already see neighboring tiles? TransMIL's position convolutions encode location but impose no edge-preserving constraint. GNN message passing such as SRMIL's is quadratic smoothing, which TVGNN argues and shows blurs boundaries (63.2% versus 53.4% clustering accuracy on Cora against MinCutPool). Neither paper reports a localization number in the file. No reported Dice or FROC result found on Lacuna for TransMIL, SRMIL or ILRA-MIL on CAMELYON16, so we measure all three ourselves.
Why not just annotate 10% of tiles? Mixed supervision does work: Dice rises from 0.215 to 0.350. But annotations are expensive and absent for rare entities. We treat the annotated model as a ceiling, and we also test whether our layer adds to it when annotations are available.
Proposed Experiments
The primary benchmark is CAMELYON16, chosen because it has exhaustive pixel annotations. We measure slide AUC, mean Dice of the thresholded heatmap, FROC, and tile-level specificity, plus runtime and peak memory at 20× and 40×.
Two decisive checks follow. The first is whether the full model's Dice approaches the annotation-assisted 0.350 while slide AUC stays within noise of the strongest rerun baseline. The second is whether the ℓ1-to-quadratic swap and the nonlocal-off ablation each measurably lower Dice. If only one term matters, the hybrid claim is false.
For generality we use TCGA-NSCLC and BRACS for slide-level metrics. TCGA-NSCLC was also used by ILRA-MIL, TransMIL and CLAM, and BRACS by SRMIL, so both allow direct comparison. Neither has localization ground truth, so there we check only that slide accuracy does not degrade, and we inspect the heatmaps qualitatively against tumor regions. A scaling study plots time and memory against tile count up to the largest slides. It confirms the linear cost and compares against TransMIL's Nyström attention on the same hardware.
References
- Attention-based Multiple Instance Learning with Mixed Supervision on the Camelyon16 Dataset
- Simultaneous Visual Data Completion and Denoising Based on Tensor Rank and Total Variation Minimization and Its Primal-Dual Splitting Algorithm
- Low-Rank Total Variation for Image Super-Resolution
- Improved total variation based image compressive sensing recovery by nonlocal regularization
- Exploring Low-Rank Property in Multiple Instance Learning for Whole Slide Image Classification
- Exploiting Label-Independent Regularization from Spatial Patterns for Whole Slide Image Analysis
- Total Variation Graph Neural Networks
- Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images
- Snuffy: Efficient Whole Slide Image Classifier
- TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification