Foveated Glimpses with a Learned Stop for Reading Gigapixel Slides

A whole-slide pathology image is a scan of a tissue section, often larger than 100,000 by 100,000 pixels. Today's slide classifiers almost always cut it into tens of thousands of small tiles, run every tile through a pretrained encoder, and only then decide what matters. Several papers train reinforcement-learning agents to "look" at only part of the slide. When they report speedups, though, the agent usually picks among feature vectors that were already computed for the whole slide, so encoding every tile, the step that dominates cost, is still paid. We propose an agent that reads the slide file directly, one glimpse at a time. Each glimpse is foveated: it is sharp at its center, read from the high-magnification level of the slide's image pyramid, and progressively coarser toward its edges, read from lower levels. The agent decides where to look next from everything it has seen so far. It also learns when to stop, so a slide with an obvious tumor costs a few glimpses and an ambiguous one costs many.

The claim we want to test is narrow. Under an honest cost accounting, counting pixels actually read from disk, encoder compute, and wall-clock time including the agent's own overhead, a sequential foveated reader with a learned stopping rule should give a better accuracy-per-cost curve than a budget-matched deterministic coarse-to-fine cascade. The advantage should be largest on slides whose diagnostic evidence is small and scattered. If a simple cascade or a greedy uncertainty heuristic matches it, then the reinforcement learning is unnecessary, and we want to find that out.

What exists and where the gap is

Three groups of work come close. The first trains agents to choose patches. RLogist trains a policy with Proximal Policy Optimization (PPO, a standard policy-gradient method) to choose grid patches after a low-magnification scan. It fixes the fraction of tissue observed in advance (20% or 50%). PAMIL has an agent select informative patches from a pre-extracted candidate pool. Both supply our sequential formulation and motivate the proposal, because the cost of encoding every patch is not removed in their evaluations. Neither uses peripheral context around a selected patch. Neither learns when to stop.

The second group uses deterministic cascades. HAG-MIL scores low-resolution patches and sends a fixed top-$k$ to higher magnification. PTree-Net builds a hierarchy of crops across three magnifications from a low-power heatmap. These methods make no sequential, history-aware decisions, and their $k$ is hand-set. They are the strongest simpler competitor and the main threat to our claim. Reinforced Auto-Zoom Net did learn a zoom-or-stop decision, but for one zoom step per region and on 10 slides. Look, Investigate, and Classify used exactly five glimpses on small fields of view, not whole slides.

The third group supplies the sensor design. FOVI samples images with density that falls off from the center, following the human cortical magnification function, a rule for how much visual cortex is devoted to each part of the visual field. It convolves over nearest-neighbor graphs on the resulting sensor surface and reaches over 96% of full-resolution ImageNet accuracy with 1/16 of the pixels. FOVI fixes its gaze, which is unusable on a slide with tissue fragments scattered across the glass. Learned saccades with scene reconstruction shows on ImageNet that a learned gaze policy beats random glimpses by 11.1%.

One paper constrains the design: Containerization on a self-supervised active foveated approach reports that sequential control loops can add enough latency to cancel the theoretical savings. We therefore report wall-clock time, not only operation counts.

Core design

A glimpse at slide coordinate $(x, y)$ reads a small fovea at 20x, a ring at 5x, and an outer ring at 1.25x. All three come from levels the scanner already stores, so no level has to be resampled at full size. The pixels are resampled onto a FOVI-style surface with locally uniform density. A pathology encoder, adapted with low-rank adapters (a small number of trainable parameters added to frozen weights) as FOVI-ViT was, maps each glimpse to a token. A transformer keeps a running slide representation, outputs a class belief, a next-fixation distribution over coordinates, and a stop probability. A one-time thumbnail pass at 1.25x masks out glass background and gives the first belief. The periphery matters because it shows the tissue context around a candidate lesion and makes the next-fixation decision better informed. We test both roles directly.

Training avoids most of the instability of sparse terminal rewards in two stages. The first stage is imitation. On training slides only, we may pre-extract features exhaustively, which is allowed as privileged information, much like the behavior-cloning-then-RL recipe in Eye, Robot. An oracle uses these features to pick, at each step, the glimpse that most reduces the slide classifier's loss. The policy first imitates this oracle. The second stage fine-tunes with PPO on the objective

$$J = \mathbb{E}\left[\log p_\theta(y \mid h_T) - \lambda\, C(T)\right],$$

where $y$ is the slide label and $h_T$ is the agent's state when it stops at step $T$. $C(T)$ is the number of high-magnification pixels read, and $\lambda$ trades accuracy against cost. Sweeping $\lambda$ traces the accuracy-cost curve. At test time no slide is ever pre-extracted.

Baselines

All baselines use the same pathology encoder, the same train/test splits, and the same hyperparameter search budget. Every method is evaluated at matched high-magnification pixel budgets of 5%, 10%, 20%, and 50% of tissue. The proposal is also evaluated at its own adaptive budget, with the average matched for comparison.

The naive reference is uniform random tile sampling at each budget with attention pooling. On CAMELYON16, where metastases often cover under 5% of tissue, this should miss small lesions, and it sets the floor.

The upper reference is exhaustive attention-based multiple-instance learning (MIL) over all 20x tiles. This is a classifier that weights each tile by a learned attention score and pools the weighted features into a slide prediction. It stands in for "pay full cost." The best exhaustive number in the file is Snuffy at AUC 0.987 on CAMELYON16. We treat this as a ceiling, not a target.

The strongest existing selective method on our question is a HAG-MIL-style cascade, rerun from BearCleverProud/HAG-MIL. Its $k$ is tuned per budget with the same search budget, and its features are computed on the fly so its costs are counted the same way as ours. HAG-MIL reports AUC 0.946 and F1 0.874 on CAMELYON16 and AUC 0.921 on TCGA-NSCLC. These figures come from exhaustive multi-resolution scoring with its own encoder, so we compare against our rerun. Among policy methods, RLogist reports CAMELYON16 AUC 0.8794 at 20% observation and 0.8874 at 50%, against CLAM-SB at 0.8809. Its inference time falls from about 40 to 9.74 minutes per slide, but on pre-extracted features. We reimplement it with on-the-fly reading at matched budgets. PAMIL reports 96.3% CAMELYON16 accuracy, also on pre-extracted ResNet50 features.

Three ablations isolate the ingredients. Fovea-only removes the 5x and 1.25x rings. Fixed-horizon removes the stop action and sets $T$ to the proposal's mean. Imitation-only drops the PPO stage.

Beating the cascade means a higher AUC at equal measured wall-clock, with a confidence interval from bootstrap over slides that excludes zero, at the 10% and 20% budgets. The 50% budget is unlikely to show a gap, because both methods saturate there.

Why not just…?

Why not just use a top-$k$ cascade with a tuned $k$? This is the real threat. A cascade ranks patches once, from a low-power view, and does not account for what it has already inspected. We expect it to spend its budget on contiguous patches of one large focus and to miss small secondary foci. It also uses a constant budget for every slide. The matched-budget comparison settles whether this matters. The stratified analysis below, which groups slides by lesion size, settles where it matters. If the cascade ties us everywhere, we will report that the sequential policy adds nothing on these benchmarks.

Why not just pick glimpses greedily by classifier uncertainty, with no reinforcement learning? A history-aware greedy rule can be strong, since it already uses everything seen so far. We include it as a baseline: at each step, fixate where the thumbnail-conditioned attention, recomputed after each glimpse, is highest, and stop when predictive entropy falls below a tuned threshold. If it matches the PPO policy, the contribution reduces to the foveated reader plus an entropy stop, which is a simpler and still useful result. The imitation-only ablation shows how much reinforcement learning adds on top of distilling the oracle.

Why not just classify a thumbnail, or run exhaustive MIL with a faster encoder? Thumbnails lose the cellular detail needed for diagnosis. PTree-Net reports 82.51% accuracy for thumbnail CNNs against 89.70% for its multi-resolution model on HER2 scoring. Exhaustive MIL with a cheaper encoder lowers the constant cost but still reads every tile. We report its wall-clock time and accuracy as a point on the same cost curve, so readers can see whether a lighter exhaustive model already dominates.

Proposed experiments

The primary benchmark is CAMELYON16. Its lesion-level annotations let us stratify slides by metastasis area and score glimpse placement with FROC, a detection metric that rewards finding each lesion while penalizing false alarms. The prediction is that our advantage over the cascade grows as lesions shrink. TCGA-NSCLC, where subtype evidence is spread across the tissue, is the contrasting case. Here the learned stop should fire early, giving cascade-level AUC at a lower cost. If the stop does not fire earlier on these slides than on small-lesion CAMELYON16 slides, the stopping claim fails. BRACS, where HMIL reports 55.56% fine-grained accuracy and poor sensitivity on rare classes, tests whether peripheral context helps subtle subtypes. We compare the fovea-only ablation to the full model on per-class sensitivity.

Three checks decide the claim. The first is the AUC-versus-wall-clock curve against the on-the-fly cascade and RLogist. The second is the correlation between glimpse count and slide difficulty: tumor area, and the oracle's loss after the thumbnail pass. The third is the periphery ablation's effect on both accuracy and how efficiently fixations are placed. All timing includes disk reads from the pyramid levels and the recurrent controller, measured on one GPU, so that controller latency cannot hide behind operation counts.