When a pathologist reads a digital slide, the viewer software already records what they do: where they pan, when they zoom into a region at high magnification, how long they stay, and where the cursor drifts. This proposal asks whether that ordinary software log can stand in for the hand-drawn tumor outlines that today's slide models need to localize cancer correctly. The log costs nothing extra to collect. The outlines cost hours of specialist time per slide. The catch is that the log records where the expert looked, not where the tumor is. Experts zoom into benign tissue that looks suspicious, and they sometimes look straight at tumor without recognizing it. We therefore propose to treat viewer telemetry as a noisy witness. Its reliability depends on whether the slide's true label agrees with the reader's verdict. Telemetry from slides with a negative ground-truth label becomes hard-negative evidence. Telemetry from slides with a positive label becomes a soft prior that the model only accepts where its own evidence agrees.
The setting is gigapixel pathology. A whole-slide image (WSI) of 100,000 × 100,000 pixels is cut into tens of thousands of tiles. A multiple instance learning (MIL) model, which predicts a slide label from an unlabeled bag of tiles, then learns which tiles matter. These models classify slides well but localize poorly. On CAMELYON16, weakly supervised CLAM reaches slide AUC 0.895 but localization Dice of only 0.215, and adding pixel masks to 10% of training slides raises Dice to 0.350 (Attention-based Multiple Instance Learning with Mixed Supervision on the Camelyon16 Dataset). Our target claim is concrete. Viewer telemetry on the training slides, with no masks at all, should recover most of that gain, and should beat a naive use of the same telemetry that treats it as ground truth.
Why this is open, and what already exists
The closest work is the PTAH testbed and the PathoGaze 1.0 dataset (Eye-Tracking, Mouse Tracking, Stimulus Tracking,and Decision-Making Datasets in Digital Pathology). It synchronized eye tracking, mouse events, and the exact image region and magnification visible on screen at every moment. It then mapped screen coordinates back into slide coordinates across zoom changes, for 19 board-certified pathologists reading CAMELYON16 slides. The scale is 171,909 fixations and more than 1.8 million mouse events. The study is descriptive and trains no model. Its findings both motivate and constrain this proposal. Pathologists show "spiky" zoom profiles that alternate between overview and cellular detail, which suggests high-magnification dwell is a targeted signal. They also commit "looked but failed to see" errors, where fixations fall inside tumor and no diagnosis follows. Their false-negative rate exceeds 75% for isolated tumor cells under 0.2 mm, while accuracy on macrometastases reaches 91%. Telemetry is therefore informative but biased, and the bias depends on lesion size.
Two lines of work supply the method. Attention-based Multiple Instance Learning with Mixed Supervision on the Camelyon16 Dataset replaces CLAM's heuristic top-k pseudo-labels with true instance labels on a fraction of slides. It also treats high-attention tiles on normal slides as negatives, which drives tile specificity from 0.864 to 0.999. We reuse that training scheme but replace its source of instance labels: telemetry instead of drawn masks. GazeRefine turns clinician fixations and dwell durations into segmentation masks by propagating from gaze-seeded prototypes in a frozen DINOv3 feature space. It reaches 88.10% Dice on Kvasir-SEG against 88.41% for fully supervised nnU-Net. That paper constrains us in the opposite direction from PathoGaze. It treats gaze as an authoritative prompt on small 512 × 512 images and needs an eye tracker. We work at gigapixel scale. We use software-only viewport and mouse logs as the primary signal. We model the reader's errors instead of trusting the reader.
RLogist is the main prior work that treats slide reading as a sequence of viewport choices. It learns with reinforcement learning which tiles to view at high magnification, from slide-level reward alone. It addresses inference cost, not localization or annotation. It never uses human trajectories. We do not try to imitate trajectories either. What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces? shows that agents cloned from expert trajectories memorize interface surface features and collapse under small perturbations: WebShop success falls from 61.2% to 4.4%. That is a reason to use telemetry as spatial evidence about tissue, not as a policy to copy.
Core Claim
The time a reader spends viewing a region at high magnification, together with revisits and cursor presence, is a cheap and spatially specific signal of suspicious morphology. Its relationship to true tumor can be untangled using two labels that already exist for every training slide: the ground-truth slide label and the reader's own verdict. A MIL model trained this way should localize metastases on CAMELYON16 about as well as one given 10% pixel masks, and much better than one trained on slide labels alone. It needs no telemetry at test time.
Research Program
The first component converts temporal UI logs into per-tile evidence. Using PTAH's screen-to-slide transform, we compute for each training tile its visibility time integrated over the session and weighted by magnification. We also record its number of distinct visits and its dwell under the cursor. Gaze dwell is computed as a separate channel for ablation only. The temporal structure matters here. A tile glimpsed at overview zoom during a pan is very different from one held at 40× for twenty seconds after a zoom spike. A static "regions visited" mask would erase that difference. We summarize these features into an exposure score $e_t$ for tile $t$.
The second component is a reliability-aware loss added to standard CLAM training. Slides fall into cases by ground-truth label and reader verdict. On ground-truth negative slides, high-exposure tiles are, by definition, benign tissue that looked suspicious to an expert. They enter as hard negatives with full weight, which extends the normal-slide rule of Tourniaire et al. to exactly the tiles most likely to cause false positives. On ground-truth positive slides, we turn exposure into a target distribution over tiles. We pull the model's attention toward it only where the model's own instance score already supports it. Concretely, we minimize
$$ \mathcal{L} = \mathcal{L}_{\text{bag}} + \lambda \sum_{t} w_t \, \ell\big(p_t, \tilde{y}_t\big), $$
where $\mathcal{L}_{\text{bag}}$ is the usual slide-level loss, $p_t$ is the model's tumor probability for tile $t$, $\tilde{y}_t$ is a pseudo-label (0 for high-exposure tiles on negative slides, 1 for high-exposure tiles on positive slides), $\ell$ is binary cross-entropy, and $\lambda$ trades off the two terms. The weight $w_t$ is the reliability of the pseudo-label. It is fixed at 1 for negative-slide tiles. For positive-slide tiles it is estimated by alternating between two steps: fitting a small logistic model that predicts the current MIL instance labels from telemetry features, then re-weighting tiles by that model's agreement with $e_t$. Reads where the pathologist called a positive slide negative are kept, not discarded. Their high-dwell tiles are exactly the "looked but failed to see" regions, and they plausibly mark tumor that was observed but not recognized.
The third component is analysis. Because PathoGaze links each read to a verdict and each slide to lesion sizes, we can measure where telemetry supervision helps. We expect it to help for macrometastases and to help little for isolated tumor cells, where readers rarely dwell. We will report this rather than hide it.
Baselines
The naive reference is weakly supervised CLAM trained only on slide labels, rerun with our patch encoder, splits, and tuning budget. For reference, Tourniaire et al. report AUC 0.895, Mean Dice 0.215, and tile specificity 0.864 on CAMELYON16 for this baseline. Every method below uses the same tile features, the same CLAM architecture, the same 270/129 split, and the same number of hyperparameter trials.
The strongest existing method on this question is mixed-supervision CLAM with true pixel masks (Attention-based Multiple Instance Learning with Mixed Supervision on the Camelyon16 Dataset). It reports AUC 0.924 and Dice 0.350 with masks on 10% of slides, and AUC 0.949 and Dice 0.405 with 80%. We rerun its 10%, 20%, and 80% settings with its own code path. We will count telemetry-only supervision as competitive if it reaches the 10%-mask Dice of 0.350 within the run-to-run spread of our reproduction, with AUC not below 0.924. We will count it as clearly better if it matches the 20% setting. We also report the cost-matched comparison, which is the practical question: telemetry on all training slides against masks on however many slides an annotator could outline in the same expert time.
The strongest alternative use of the same telemetry is a GazeRefine-style pipeline (GazeRefine). We seed foreground and background prototypes from high-exposure tiles in frozen tile features, propagate by k-nearest-neighbor affinity across each training slide, and feed the resulting pseudo-masks into mixed-supervision CLAM as if they were real masks. This tests whether training-free propagation is enough without a reliability model.
Three ablations isolate the key ingredient. The first is naive telemetry: exposure used directly as an attention target with $w_t = 1$ everywhere and no distinction between negative and positive slides. The second is shuffled telemetry: each slide receives the exposure map from a different slide, restricted to its own tissue mask. This tests whether the gain is just a tissue-density prior. The third is static telemetry: a binary visited/not-visited map with no dwell, magnification, or revisit information. This tests whether the temporal dynamics carry the signal. Further ablations drop gaze versus keep it, use correct reads only versus all reads, and vary the number of readers per slide.
Why not just…?
Why not just pay for masks on 10% of slides? This works, and it is the bar we must meet. Its weakness is cost and scale, not accuracy. Each mask means a pathologist stops clinical work to trace boundaries. Viewer logs are produced during reads that happen anyway. The cost-matched experiment settles whether telemetry from every slide beats masks on the few slides the same budget would buy. If masks on 10% still win clearly at matched cost, we will report that telemetry is not worth it for this task.
Why not just use a stronger self-supervised tile encoder with plain slide-label MIL? Better features may raise slide AUC, but the localization failure comes from which tiles get credit for the label, not only from feature quality. The 0.864 tile specificity of weak CLAM reflects attention landing on benign tissue. Nothing in the file shows that encoder changes alone fix this. To test it directly, we run every method on two encoders. If the telemetry gain shrinks to nothing with the stronger encoder, the proposal fails on that point.
Why not just prompt from telemetry at test time, the way GazeRefine prompts from gaze? Test-time prompting needs a pathologist to read every new slide first, which defeats automated screening. It also inherits reader misses, including the more than 75% miss rate on isolated tumor cells. Using the same propagation offline to create training pseudo-masks is a real threat, and it is included as a baseline. If it matches our reliability-weighted loss, the simpler pipeline wins and we will say so.
Why not just learn where to look with reinforcement learning, as RLogist does? RLogist solves a different problem: reading fewer tiles while keeping slide AUC. It reports 0.8794 AUC at 20% observation against 0.8809 for exhaustive CLAM-SB, with time per slide falling from about 40 to 9.74 minutes. It produces no localization supervision. Imitating human trajectories to build such a policy risks the interface shortcutting documented for trajectory fine-tuning. We use telemetry only as spatial evidence about tissue. A follow-up could seed an observation policy with telemetry-trained attention, but that is outside this proposal.
Proposed Experiments
All experiments use CAMELYON16 with PathoGaze 1.0 telemetry. Exp. II covers 9 pathologists across 397 slides and Exp. I covers 10 pathologists across 60 slides. Telemetry is used only for the 270 training slides and never for the 129 test slides. Tiling and feature extraction follow the CAMELYON16 pipelines in BearCleverProud/HAG-MIL and coglabiisc/SASHA. Metrics match Tourniaire et al.: slide AUC, Mean Dice against reference tumor masks, and tile specificity. Dice is additionally broken down into macrometastases, micrometastases, and isolated tumor cells.
Three checks decide the outcome. The first is whether telemetry-only training reaches the 10%-mask Dice of 0.350 without lowering AUC. The second is whether it beats naive telemetry and shuffled telemetry by a margin larger than seed variance. If shuffled telemetry does as well, the signal is only a tissue prior and the claim fails. The third is whether mouse-and-viewport telemetry alone keeps most of the gain over the version with gaze, which determines whether the method can run in any clinic without eye trackers.
A secondary check addresses reader idiosyncrasy. We train with telemetry from a subset of pathologists and confirm that the result does not depend on which readers are held out. This guards against the model learning one person's scanning habits rather than tissue appearance.
References
- Attention-based Multiple Instance Learning with Mixed Supervision on the Camelyon16 Dataset
- Eye-Tracking, Mouse Tracking, Stimulus Tracking,and Decision-Making Datasets in Digital Pathology
- GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation
- RLogist: Fast Observation Strategy on Whole-Slide Images with Deep Reinforcement Learning
- What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?