Correcting Where Pathologists Looked: Debiased Gaze Supervision for Whole-Slide Attention

Pathologists' eye movements are an attractive free label for training slide-reading AI. When a pathologist zooms in and lingers on a patch of tissue, that is a cheap hint that the patch matters. The catch is the one recommender systems learned the hard way. A click log only records items the interface chose to show, so an unclicked item is not evidence of disinterest. In the same way, a gaze log only records tissue the viewer software and the pathologist's habits brought into view. Where the slide opens, how the viewer centers it, and which lesions are big enough to catch at low zoom all decide what gets fixated, apart from what is actually there. If we train a model on raw gaze, it learns that unvisited tissue is uninteresting. That is exactly the wrong lesson for the small, peripheral, boundary-hugging structures that matter most in tumor microenvironment profiling, the study of the immune and stromal cells around a tumor, where an immune cell's prognostic meaning depends on whether it sits inside a tumor nest or in surrounding stroma.

We propose to treat pathologist gaze on gigapixel whole-slide images (WSIs) as a presentation-biased feedback log. We would explicitly estimate the probability that each tissue region was examined at all, as a function of how it was presented, and then use inverse-propensity-corrected gaze as auxiliary supervision for the attention map of a multiple-instance learning (MIL) slide classifier. MIL here means a model that sees a slide as a bag of tiles and learns which tiles drive the slide-level label. The claim is narrow and testable. Naive gaze supervision should sharpen localization of large, centrally placed lesions but fail on, or even suppress, small and peripheral ones. Propensity correction should recover them, at a tiny fraction of the annotation cost of pixel-level labels.

Core Claim

The obvious bridge between these fields, using gaze to supervise attention, has not been done in the file's literature. Its biased form, however, is the default anyone would build first. PathoGaze 1.0 records 19 pathologists reading CAMELYON16 slides, with fixations, saccades, mouse and viewport logs mapped into slide coordinates. It documents strongly scale-dependent search. Accuracy is 91% on macrometastases but the error rate exceeds 75% on isolated tumor cells, and pathologists scout peripheral tissue with the mouse while their eyes stay central. That paper supplies the data and motivates the gap, but it proposes no learning algorithm and no bias correction. On the presentation-bias side, Counterfactual Augmentation for Multimodal Learning Under Presentation Bias shows that treating "what was shown" as a mediator, and correcting for it, matters for models trained on interaction logs. We borrow that framing, not its generative model.

Our specific contribution is a presentation-aware examination model for WSI viewing. It separates "was this region ever examined at diagnostic magnification" from "does this region attract attention once examined," and plugs the first quantity into a debiased auxiliary loss on MIL attention. Let $g_r \in \{0,1\}$ indicate that region $r$ received at least a threshold of high-magnification fixation dwell. Let $x_r$ be presentation covariates of that region: distance from the initial viewport center, distance to the slide's tissue edge, screen position when first in view, and connected-tissue size at thumbnail scale. Let $y_r$ be the unknown relevance. Borrowing the examination hypothesis from click models, we assume

$$P(g_r = 1) = \theta(x_r)\,\gamma(y_r),$$

where $\theta$ is the examination propensity, depending only on presentation, and $\gamma$ is the probability that an examined region attracts dwell, depending on content. With an estimate $\hat\theta_r$, the MIL model's tile score $s_r$ is trained with the unbiased positive-unlabeled loss

$$\mathcal{L}_{\text{gaze}} = -\sum_r \Big[\tfrac{g_r}{\hat\theta_r}\log\sigma(s_r) + \big(1-\tfrac{g_r}{\hat\theta_r}\big)\log\big(1-\sigma(s_r)\big)\Big],$$

where $\sigma$ is the sigmoid and $\hat\theta_r$ is clipped from below to control variance. This loss is added to the usual slide-level classification loss. Setting $\hat\theta_r$ to a constant recovers ordinary positive-unlabeled learning. Setting $\hat\theta_r = 1$ recovers naive gaze supervision. These two special cases are our key ablations.

Research Program

The first stage builds region-level gaze labels from PathoGaze. Fixations are mapped to 256-pixel tiles at the slide's diagnostic magnification using the dataset's screen-to-slide registration and Otsu tissue masks, and dwell is accumulated per tile per reader. The ten readers in Experiment I all viewed the same 60 slides. That gives repeated readings of identical tissue under varying navigation paths, so the same region is presented differently across readers. This variation lets us identify $\theta$ separately from relevance, playing the role that randomized result swaps play in search-engine debiasing.

The second stage estimates $\theta$ in two ways. The primary estimator is expectation-maximization over the factorization above, with $\theta$ a small gradient-boosted model on $x_r$ and $\gamma$ a function of frozen tile embeddings. It uses no lesion annotations. As an oracle upper bound only, we also fit $\theta$ on a held-out calibration subset of training slides where CAMELYON16 pixel annotations reveal the true relevance. The gap between the two estimators tells us how much our method loses by not knowing ground truth.

The third stage trains a context-aware MIL backbone with the auxiliary loss on training slides that have gaze, and evaluates on CAMELYON16 test slides with no gaze used at test time. The backbone is the CAMIL architecture, a Nystromformer for slide-wide context plus attention constrained to each tile's 8 spatial neighbours. Gaze is also excluded from any test slide that a reader viewed during training.

Two recognized failure modes constrain the design. PathoGaze's "looked but failed to see" errors mean that dwell does not equal correct recognition. Gaze labels therefore mark tissue judged worth inspecting, which includes benign mimics, so $\gamma$ is not assumed to equal relevance. We report how much tissue that readers dwelled on but annotations mark benign. Mixed supervision on Camelyon16 shows that weakly supervised attention leaks into healthy tissue (specificity 0.864). Our naive-gaze arm could make that worse by rewarding benign-but-salient regions, which is why tile-level specificity is a primary metric.

The tumor-microenvironment direction enters as the evaluation regime and the reason this matters, not as the dataset. Stabilin-1+ macrophage microlocalization in bladder cancer found that only intratumoral, not stromal, macrophages predicted outcome. The K17 tumor-immune spatial framework quantifies immune cells inside a 100 µm influence band around tumor nests. Both say the clinically meaningful signal lives at small scales and at tumor boundaries, which are precisely the places a presentation-biased gaze log underexposes. We therefore score localization separately inside a 100 µm band around annotated tumor boundaries, and stratified by lesion size and by distance from the initial viewport. CAMELYON16 is lymph-node metastasis rather than immune-cell profiling. The file contains no gaze data on multiplex-stained TME slides such as the 23-slide pancreatic cohort, so transfer to true immune-niche profiling is the stated next step, not a claim of this work.

Baselines

The naive reference point is CLAM-style attention MIL trained on slide labels only. The mixed-supervision paper reports a Camelyon16 Dice of 0.215, specificity 0.864 and AUC 0.895 for it, and that is the floor.

The strongest existing method without human signal is CAMIL. It reports Camelyon16 AUC 95.9% and Dice 0.515, against 0.459 for CLAM-SB and 0.103 for TransMIL. Because our method uses the CAMIL backbone, any gain is attributable to the gaze loss rather than to architecture. Published Dice values across these papers may use different thresholds and protocols, so we re-run CLAM, CAMIL and all our arms under one Dice protocol, one patch encoder, one train/test split and an identical hyperparameter search budget. Beating CAMIL means a statistically reliable Dice gain over five seeds with no AUC loss. The gain must be concentrated in the micrometastasis and boundary-band strata.

The strongest human-signal competitor is mixed supervision with pixel annotations. It reports Dice 0.350 and specificity 0.999 with 10% annotated slides on CLAM, and Dice 0.405 at 80%. We re-implement it on the CAMIL backbone at 5%, 10% and 25% annotated slides. Success here is matching the 10%-annotation arm's Dice using gaze alone. Gaze is a byproduct of routine reading, so matching would be a real cost win even with no absolute win.

The ablations isolating the key ingredient are the auxiliary loss with $\hat\theta = 1$ (naive gaze), with constant $\hat\theta$ (uniform positive-unlabeled learning), with our estimated $\hat\theta$, and with the annotation-calibrated oracle $\hat\theta$. The proposal's thesis holds only if the estimated propensity beats both naive and uniform versions specifically on small and peripheral lesions.

Why not just…?

Why not just supervise attention with raw gaze dwell? This is the default and the main object of study, so it is a full experimental arm. PathoGaze shows that fixation density tracks lesion size and central placement, with ITC error above 75%. Raw gaze therefore labels most isolated tumor cells and peripheral foci as negatives. We predict naive gaze will match or beat our method on macrometastases and lose on ITCs and the boundary band. If it does not lose there, the presentation-bias thesis is wrong for this setting and we will say so.

Why not just treat gaze as positive-unlabeled data with a constant labeling rate? This is the most serious threat. Our loss is a positive-unlabeled loss, and the only difference is whether the labeling rate varies with presentation. If readers' viewing were close to uniform over tissue, the uniform version would do as well. The constant-$\hat\theta$ ablation settles this directly. We also report how much $\hat\theta$ actually varies across slides as a diagnostic: if its spread is small, we expect, and would accept, no difference.

Why not just buy pixel annotations for a few slides? Mixed supervision shows that 10% annotations already lift Dice from 0.215 to 0.350, with diminishing returns to 80%. This is a strong, simple alternative. Our case rests on gaze being free and already collected, so the fair comparison is the annotation-budget curve. Annotations also inherit the same perceptual blind spots for tiny lesions that PathoGaze documents. If 5% annotations dominate debiased gaze at every stratum, the practical argument for gaze collapses, and the contribution reduces to the measurement of viewing bias itself.

Why not just use the counterfactual GAN from the recommender work instead of propensities? Counterfactual Augmentation did beat IPW on Airbnb and clothing data. Its own limitations note, however, warn that synthetic labels can poison training when the generator mismatches real behavior. Synthesizing plausible gaze over gigapixel tissue is much harder than synthesizing a rating. With clipped propensities and the oracle comparison, we can measure whether variance, the usual weakness of inverse weighting, is actually the bottleneck before paying for a generator.

Proposed Experiments and Decisive Checks

All arms are trained on CAMELYON16 training slides with PathoGaze logs and evaluated on the 129 test slides with pixel annotations. Metrics are slide AUC, tile-level Dice and specificity, and Dice stratified by lesion size (macrometastasis, micrometastasis, isolated tumor cells), by distance from the reader's typical initial viewport, and within the 100 µm tumor-boundary band. Multi-center robustness is checked on CAMELYON17 slide-level AUC, where CAMIL reports 88.1%, using models trained on CAMELYON16 gaze.

The decisive check is a crossover. Naive gaze should help most where readers already look and fail where they do not, and estimated-propensity gaze should close the small-lesion and boundary gap toward the oracle-propensity arm. A secondary check uses Experiment I's shared slides. We hold out readers and ask whether $\hat\theta$ fitted on some pathologists predicts examination by others. If it does, viewing bias is a stable property of the viewer and the task rather than of individuals, and the propensity map itself becomes a usable audit of which tissue a given interface systematically hides.