Remembering What Was Ruled Out: Residual-Risk Memory for Whole-Slide Search

A pathology slide scanned at full resolution is a gigapixel image, roughly 100,000 by 100,000 pixels. The lesion that decides the diagnosis, such as a few tumor cells in a lymph node, may cover under 1% of it. Current systems either compute features for every small tile, which takes about 40 minutes per slide according to RLogist, or let a learned policy look at a chosen subset. Those policies stop at an arbitrary budget and keep no record of how well each region has already been checked. We propose to give a slide-reading agent the kind of persistent spatial memory that robot navigators use: a compact map of the slide. For every tissue cell, the map records the coarsest magnification at which it has been inspected and the current probability that it still hides a lesion. That map then answers two questions: where to look next, and when it is safe to stop.

The central point is that "looked at" is not the same as "ruled out." A region that looked benign at 5x magnification is strong evidence against a large metastasis but weak evidence against isolated tumor cells. A memory that stores only visited locations cannot express this. The agent reads the slide by repeatedly picking the cell whose inspection most reduces the probability that a lesion remains anywhere unseen, possibly far from where it currently is. It halts when that probability falls below a clinically chosen threshold. If this works, slide reading would go from a fixed observation budget to a stated guarantee about misses. The largest benefit should come exactly where current selective-zoom methods fail, on slides whose only tumor is very small.

Core Claim

We claim that the important missing ingredient in active whole-slide reading is a memory of residual risk per region, conditioned on the resolution at which it was examined. We do not claim that memory of visited places is the missing piece. Concretely, an agent that keeps a calibrated, resolution-aware belief map, and updates it as high-magnification evidence arrives, should match exhaustive reading on slide-level accuracy while reading a fraction of the tissue. It should also recover far more micro-metastases and isolated tumor cells than either static thumbnail ranking or greedy zooming at equal compute. A further consequence is a stopping rule with a meaning: stop when the estimated probability of a missed lesion is below $\epsilon$.

Write the slide as a grid of cells $c$. Let $p_{c,t}$ be the agent's probability, after $t$ inspections, that cell $c$ contains a lesion. Here $p_{c,t}$ is computed from the observations made so far and from the known sensitivity of the patch detector at the magnification each cell was seen at. The slide-level residual risk is then

$$R_t = 1 - \prod_c (1 - p_{c,t}),$$

which is the probability that at least one unseen lesion remains, under a cell-independence approximation. The agent picks the next action, a cell together with a magnification $m$, by expected risk reduction per unit cost:

$$a_{t+1} = \arg\max_{(c,m)} \frac{\mathbb{E}[R_t - R_{t+1} \mid c, m]}{\text{cost}(m)},$$

where $\text{cost}(m)$ is the number of pixels read at magnification $m$. It stops when $R_t < \epsilon$. The independence assumption is crude, because lesions cluster. We handle clustering by having a positive finding raise neighbors' priors, and we measure in our experiments whether the resulting $R_t$ is calibrated.

Research Program

The memory itself borrows from autonomous driving. HRMapNet showed that storing past perception as a small rasterized semantic grid, about 1 MB per kilometer, rather than as high-dimensional feature tensors, keeps a city-scale memory cheap while still guiding attention. We use the same idea as a multi-channel canvas over the tissue mask. The channels hold the finest magnification inspected, the current lesion probability, and detector confidence. This avoids the memory growth of archives that store full embeddings for every state, which the file flags in Intelligent Go-Explore and in graph planners. Because the selection rule is global over the canvas, the agent can return to any earlier frontier, at any magnification, in a single step. This is the "graph jump" that the Evolving Graphical Planner uses to correct navigation mistakes. Here it comes from the belief map rather than from a learned graph network, which keeps the planner simple. Backtracking across magnification levels therefore needs no extra machinery. That is precisely what feedforward zoom methods such as MagNets and PTree-Net lack: their early zoom decisions cannot be undone.

The decomposition follows MPS-VAS, which split visual active search on satellite imagery into a task-specific predictor that updates a target heatmap from inspected locations and a separate search policy. It reported 15–35% better target retrieval than monolithic end-to-end reinforcement learning under distribution shift. That paper supplies our design and motivates avoiding pure reward-driven training. It also constrains us, because it assumes every query returns a noiseless label at a single resolution. In pathology, inspections are noisy and their reliability depends on magnification and lesion size, so that assumption fails. Our main technical contribution is the observation model that replaces the oracle. For each magnification, we fit detector sensitivity as a function of lesion size using the exhaustive pixel annotations in CAMELYON16. A negative finding at magnification $m$ then lowers $p_{c,t}$ only by the amount that detector sensitivity justifies.

Calibration is the main risk, and prior work tells us where it breaks. Tourniaire et al. showed that clustering-constrained attention (CLAM) attention maps often flag benign tissue, with tile specificity of 0.864. Adding tile annotations to 10% of slides raised specificity to 0.999 and localization Dice from 0.215 to 0.350. Attention weights are therefore not probabilities. We train the patch detectors with the same mixed supervision and then recalibrate them per magnification on held-out annotated slides. The spatial prior update, in which tumor found in one cell raises its neighbors' probability, plays the role of RLogist's conditional feature super-resolution. The difference is that it acts on explicit probabilities rather than latent features, so its effect on $R_t$ can be inspected directly.

We considered and rejected a self-repellent occupancy term of the kind analyzed in Doshi et al. as the main mechanism. Penalizing revisits by $(x_j/\mu_j)^{-\alpha}$ provably reduces sampling variance, but it pushes the agent toward uniform coverage. Our goal is targeted risk reduction instead. We keep it only as an ablated alternative routing rule.

Baselines

The naive reference point is exhaustive reading: features from every tile at 20x, aggregated by single-branch CLAM. RLogist reports this at AUC 0.8809 on CAMELYON16, at roughly 40 minutes per slide. We rerun it with the same feature extractor used in every other arm, so that only the reading strategy differs. Beating it means matching its AUC within confidence intervals while reading far fewer high-magnification pixels.

The strongest aggregation baseline is dual-stream MIL (DSMIL). It scores every tile, takes the highest-scoring tile as an anchor, and weights all other tiles by their learned distance to it. DSMIL reports 89.9% accuracy and a localization FROC of 0.437 on CAMELYON16. Its FROC is the bar for our lesion-level detection, since it sees everything.

The strongest active-reading competitor is RLogist, a reinforcement-learning policy trained with proximal policy optimization that starts from a thumbnail and selects high-resolution tiles. It reports AUC 0.8874 at 50% observation and 0.8794 at 20%, with 9.74 minutes per slide at 20%. We run it with its published budgets and also at the exact pixel budget our agent chooses on each slide, and we give both methods the same detector features and tuning trials. Beating it means higher AUC, or equal AUC at lower average tissue read, and above all higher micro-metastasis recall at matched budget.

For lesions that a coarse view misses, the reference is MagNets. It reports CAMELYON17 AUROC of 84% excluding isolated-tumor-cell cases but 71% (3-layer) and 75% (4-layer) on all cases, while reading 3–6 million pixels per slide. Closing that gap at a comparable pixel count is the clearest win available.

Three ablations isolate the key ingredient. The first, a static ranking baseline, scores all cells once on the thumbnail and reads them in order, using the same detectors and the same $R_t$ stopping rule but never updating beliefs. The second is a visited-mask memory that records where the agent has looked but not at what magnification or with what residual probability. The third is a version without the resolution channel, which treats any inspection as fully informative. We also compare our stopping rule with fixed budgets at matched average cost, and the risk-reduction router with a self-repellent router.

Why not just…?

Why not just rank regions once on the thumbnail and read the top k? This is the real threat. On slides with large metastases, a single low-magnification ranking probably finds the tumor immediately, and a dynamic memory adds nothing. The evidence that it falls short on small lesions is indirect. MagNets, which also commits on coarse attention, loses 13 AUROC points when isolated tumor cells are included. PathNavigate's surprise field, a single low-magnification pass, is reported to struggle beyond focal lesions. We do not assume the answer. The static-ranking ablation shares our detectors and stopping rule, so if it matches us on the small-lesion subset, the memory is unnecessary and the contribution shrinks to the calibrated stopping rule. We would report it as that.

Why not just add a visited mask to RLogist? Recording where the agent has been removes redundant revisits. That is what action history did in navigation, where Think, Remember, Navigate found that removing a 10-action memory dropped HM3D success from 54.3% to 44.0%. But a slide reader that never re-inspects already avoids loops. Its failure is in treating a 5x glance as a full check. The visited-mask ablation tests exactly this. Our prediction is that it matches us on macro-metastases and falls behind on micro-metastases.

Why not just extract everything and use a long-context aggregator? MambaMIL+ aggregates 10,000+ tiles in linear time and reports about 1.5% average relative AUC gains. It still requires features for every tile up front, so the 40-minute extraction cost remains. Exhaustive aggregation is also the accuracy ceiling we aim to match, not exceed. If compute were free this alternative would win, and the comparison with exhaustive CLAM and DSMIL makes that trade-off explicit.

Proposed Experiments

The primary study uses CAMELYON16. We train on its 270 training slides with tile annotations on 10% of them, holding out annotated slides for per-magnification calibration, and test on its 129 test slides. We report slide AUC, FROC, high-magnification pixels read, wall-clock time, and the empirical miss rate against the nominal $\epsilon$. The second study transfers to CAMELYON17 without retraining and stratifies results into macro-metastasis, micro-metastasis, and isolated-tumor-cell slides, matching the MagNets split. TCGA-NSCLC subtyping (LUAD vs. LUSC) serves as a diffuse-pattern control. There, residual risk of a missed focal lesion is the wrong objective, and we predict the method offers no advantage over static ranking. For reference, the GNN of Adnan et al. reports 88.8% accuracy and AUC 0.89 on this task. A positive result on TCGA-NSCLC would make us suspect that we were measuring something other than intended.

The decisive check is a curve of micro-metastasis and isolated-tumor-cell recall against high-magnification pixels read, on the CAMELYON17 small-lesion strata. It compares the full agent, static ranking, visited-mask memory, the no-resolution-channel variant, RLogist, and MagNets. The proposal succeeds if the full agent sits strictly above static ranking and visited-mask memory on this curve, while its slide AUC on CAMELYON16 is statistically indistinguishable from exhaustive CLAM. A second decisive check is calibration. Across test slides, the fraction of slides where the agent stopped yet an annotated lesion lay in an unread or under-resolved cell should not exceed $\epsilon$ by more than its confidence interval. If this check fails, the stopping rule has no safety meaning and the cell-independence approximation must be replaced before any clinical framing is defensible.