Spending High-Resolution Compute Where Pathologists Miss

A whole-slide pathology image is a gigapixel scan of tissue. Finding a cluster of a few tumor cells in it is a needle-in-a-haystack problem. A cheap low-resolution pass blurs away the smallest targets. A full-resolution pass over every tile is slow. Coarse-to-fine detectors split the difference: a low-resolution screening stage picks a small fraction of the slide to re-examine at full magnification. Today those screeners are trained to find where tumor is likely. In the clinic, though, the AI is usually a second reader beside a pathologist, and the pathologist already catches most large lesions. We propose training the screener to send its limited high-resolution budget to likely tumor that a human reader would probably miss. The estimate of "would probably miss" comes from pooled eye-tracking of many pathologists reading the same slides.

The opportunity is concrete. A recent eye-tracking study of 19 pathologists reading lymph-node slides, PathoGaze 1.0, reports 91% accuracy on large metastases but error rates above 75% on isolated tumor cells (deposits under 0.2 mm). It also records many "looked but failed to see" errors, where the reader's gaze landed on the lesion and the reader still called it benign. Those failures are concentrated, structured, and measurable. A detector that knows where they cluster can be built to cover them on purpose rather than by accident.

Core Claim

Under a fixed budget of high-resolution compute, a coarse-to-fine metastasis detector reaches higher combined sensitivity on small lesions when its screener ranks regions by "probability of tumor times probability the human misses it." Combined sensitivity counts a lesion as found if either the pathologist or the AI finds it. The comparison is against a screener that ranks by tumor probability alone, at the same false-positive rate. The human-miss probability is learned from multi-observer gaze that is aggregated separately at each zoom level. A region counts as inspected only if readers fixated it at a magnification high enough to resolve small lesions. Gaze is needed only at training time. At deployment the screener predicts human coverage from the low-resolution image, in the spirit of Deep Learning For Inter-Observer Congruency Prediction. That work showed plain convolutional features predict how much observers will agree on an image, with Pearson r = 0.611 on Judd/MIT. We extend this from one score per image to a map over the slide.

The ranking rule follows from a simple budgeted objective. Let $r$ index candidate regions at low resolution, $p(r)$ the probability that $r$ contains tumor, $h(r)$ the probability that a reader detects it, and $B$ the number of regions we can afford to re-examine at full resolution. The expected number of lesions caught only by the AI is maximized by choosing

$$ S^\star = \arg\max_{|S|\le B} \sum_{r\in S} p(r)\,\big(1-h(r)\big), $$

so the screener should rank regions by $p(r)(1-h(r))$. The second factor is the new ingredient, and the paper is about whether it actually helps.

Relation to Prior Work

Three groups of prior work bear on this.

The first group supplies the method. SaccadeDet runs a low-resolution density regressor that discards about 95% of a gigapixel image, then runs a full-resolution detector on what remains. On Camelyon slides it cuts processing to 2.1–2.5 minutes per slide. Its "saccade" and "gaze" stages are algorithmic names only, and they use no human data. We keep its two-stage structure and change only the screener's ranking target. Exhaustive sliding-window classification, as in Detecting Cancer Metastases on Gigapixel Pathology Images, and spatial smoothing with a conditional random field, as in Neural Conditional Random Field, set the accuracy reference points that a budgeted system must approach.

The second group motivates the proposal. PathoGaze 1.0 shows that human failure on small lesions is common and comes in two kinds: never looking (a search failure) and looking without recognizing (a recognition failure). The paper offers no algorithm for pooling its readers' scanpaths into a training signal.

Training a network to attend like human drivers is the closest precedent for using pooled gaze against class imbalance. It upweights whole video frames whose pooled driver gaze departs from the usual center bias, reaching a KL divergence of 1.24 on the BDD-A driving benchmark versus 1.95 for DR(eye)VE. Our use differs in three ways. Our weighting is spatial rather than per frame. It is resolved by magnification. It targets where humans fail rather than where humans look.

Supporting Mitosis Detection AI Training with Inter-Observer Eye-Gaze Consistencies turns pooled gaze from 14 lay observers into positive labels, reaching 0.83 average precision against 0.85 for expert labels. That approach treats fixation as evidence of a target, so it would inherit exactly the failures we want to cover.

The third group constrains the design. Saliency Revisited finds that mouse movements agree across viewers much less than fixations do: a congruency score (shuffled AUC) of 0.61 versus 0.76. PathoGaze notes that pathologists often move the cursor away from where they are looking. We therefore build the coverage map from fixations and use cursor and viewport data only to recover the zoom level. The expert-versus-novice gaze differences in surveillance footage caution against substituting lay observers, so the human-miss model is trained only on the pathologist cohort.

Research Program

Building the coverage map. PathoGaze already maps screen-space fixations into slide coordinates and records the zoom level at each moment. For each fixation we draw its foveal footprint in slide space. We keep only fixations made at or above a magnification threshold, chosen by checking at what zoom isolated tumor cells become separable in the reference annotations. Pooling across readers gives, for each region, the fraction of readers who inspected it at sufficient resolution.

Combining coverage with reader decisions. We pair the coverage map with each reader's diagnostic decision on the slide. This lets us estimate a per-region detection probability $h(r)$ for tumor regions, and to separate "not looked at" from "looked at but failed." If PathoGaze provides only slide-level decisions rather than lesion-level ones, $h(r)$ will be approximated. A tumor region counts as detected when it was fixated at sufficient magnification on a slide the reader called positive. We will report how sensitive the results are to this approximation.

Training the screener. The screener is a low-resolution network with two heads. One predicts $p(r)$ from the Camelyon16 tumor masks. The other predicts $h(r)$ from the image alone, trained on readers whose data are kept separate from evaluation. The full-resolution stage is identical across all systems: an ImageNet-pretrained patch classifier with hard negative mining, following Liu et al.

Analysis. Beyond accuracy, we check whether the predicted $h(r)$ captures something besides lesion size and stain contrast. We will measure how much of its variance those two simple predictors explain, and how much remains for things like tissue context or nearby lymphocyte-dense look-alikes.

Baselines

Unaided pathologist. The naive reference is the pathologist alone. Liu et al. report 73.2% sensitivity for an exhaustive human search, and PathoGaze reports above 75% error on isolated tumor cells. We recompute human sensitivity, broken down by lesion size, for the held-out PathoGaze readers on our evaluation slides.

Exhaustive sliding window. The strongest accuracy reference is the exhaustive classifier of Liu et al., which reports 92.4% sensitivity at 8 false positives per slide on Camelyon16. We treat it as the unbudgeted ceiling, using the same full-resolution classifier as our second stage. NCRF, which reports an average FROC of 0.8096, is run as an additional ceiling with spatial smoothing.

Tumor-only screener. The strongest direct competitor is a SaccadeDet-style screener that ranks by $p(r)$ alone. It is the system practitioners would actually deploy under a compute budget. We found no reported result on Lacuna for SaccadeDet's detection accuracy on Camelyon16, since the file gives only its runtime, so we retrain it ourselves. It uses the same backbone, the same second stage, the same training slides, and the same hyperparameter search budget as our method.

Size-reweighted screener. This ablation isolates the gaze ingredient. The screener ranks by $p(r)$ times a weight that grows as lesion size shrinks, fitted from the masks to mimic the human error curve by size.

Uncertainty-routed screener. A second ablation replaces $1-h(r)$ with the screener's own predictive uncertainty.

Oracle variant. We also test using the actual reading pathologist's recorded gaze instead of the predicted coverage, to bound what a gaze-equipped viewer could give.

What counts as winning. Our method beats the competitors if it improves combined sensitivity on isolated tumor cells and micrometastases at matched budget and matched false positives per slide. The improvement must hold across held-out readers and resampled slides, and it must not lose more than a small pre-registered margin on macrometastases.

Why not just…?

Why not just train the screener for standalone sensitivity and let complementarity happen? This is the main threat, and the tumor-only screener exists to test it. If the screener's budget is large enough to cover nearly all likely tumor, ranking by human-miss probability changes little, and the two systems will tie. The idea only matters when the budget binds. We therefore sweep the budget from a few percent of tissue up to exhaustive coverage. We expect any advantage to appear at small budgets and vanish at large ones. If no budget shows a gap, the claim is false.

Why not just upweight small lesions, since that is where humans fail? This is a serious alternative. PathoGaze's size-stratified error rates suggest that size alone may explain most human misses. The size-reweighted screener tests this directly. Gaze earns its cost only if two things hold. First, $1-h(r)$ must beat the size weight on combined sensitivity. Second, a residual of $h(r)$, after regressing out size and contrast, must predict reader misses on held-out readers. If gaze adds nothing beyond size, we will report that as the finding: a mask-derived size prior is then sufficient, and gaze collection is unnecessary.

Why not just use the model's own uncertainty to decide where to zoom? Model uncertainty marks regions that are hard for the network. Our target is regions that are hard for the human. These two sets can coincide or diverge, and the uncertainty-routed screener measures which. The "looked but failed to see" cases in PathoGaze suggest they diverge on look-alike tissue that the network may classify confidently. We will report the overlap between the two region sets as well as their effect on combined sensitivity.

Proposed Experiments and Decisive Checks

All experiments use Camelyon16 slides covered by PathoGaze 1.0. We split PathoGaze readers so that no reader whose gaze trained the human-miss head is used to measure human performance in evaluation, and we keep training and evaluation slides disjoint.

Metrics. We report three quantities at fixed false positives per slide: combined (human-or-AI) lesion sensitivity broken down by size, the AI's standalone FROC, and the fraction of tissue processed at full resolution.

Check 1: budget sweep. This is the decisive experiment. It plots combined sensitivity on isolated tumor cells against the high-resolution budget for our screener, the tumor-only screener, the size and uncertainty ablations, and the gaze oracle.

Check 2: what the human-miss predictor learns. We test whether the image-only $h(r)$ head generalizes to held-out pathologists. We measure this by how well it ranks those readers' actual misses (AUC), and how well it does after size is regressed out.

Check 3: standalone cost. We confirm that the AI's own FROC does not fall below the tumor-only screener by more than the pre-registered margin. A second reader that helps the team but is markedly worse alone would be hard to deploy.

Expected outcome and interpretation. We expect our screener to beat the tumor-only screener at tight budgets on small lesions, and the gap to shrink as the budget grows. The interpretation hinges on the size ablation. If gaze beats the size prior, multi-observer gaze carries real information about where expert reading fails. If it does not, the work still delivers a budgeted second-reader objective and a negative result on the value of gaze.