A whole-slide pathology image has billions of pixels. Fast systems therefore look at a blurry low-magnification overview first and zoom in only where that overview looks suspicious. This works until the disease is exactly what the blur erased: a small cluster of tumor cells with no contrast at low magnification gets a low score, is never zoomed into, and is lost for good. We propose a second signal for deciding where to zoom. Instead of asking "does this region look abnormal at low resolution?", it asks "is this region one where the low-resolution view is a poor predictor of what high resolution would show?" We estimate that second quantity while a slide is being read, using online dictionary learning, a classic method that summarizes a stream of examples with a small set of reusable patterns ("atoms") plus a few running summary matrices. It costs constant memory no matter how many tiles stream past.
The core idea fits in two sentences. While a slide is read, a few zoomed-in tiles teach a pair of linked dictionaries how low-magnification appearance maps to high-magnification appearance, and running statistics record which low-resolution patterns turned out to hide the most unpredictable high-resolution detail. Unvisited tiles built from those "deceptive" patterns get extra zoom budget, even when an attention model finds them unremarkable.
Core Claim
Coarse-to-fine slide readers fail in a specific way. Selection by low-magnification attention throws away regions that look benign but are ambiguous at that magnification. Measuring ambiguity is a different problem from measuring abnormality, and it can be estimated per slide, cheaply, from the sufficient statistics that online dictionary learning already maintains. If we allocate part of a fixed zoom budget by this estimated "downsampling residual," recall of small lesions should rise at the same patch budget. The prediction is sharpest for Camelyon16 micrometastases, where low-resolution saliency is weakest.
Where This Sits
The irreversible-pruning problem comes from three coarse-to-fine systems, and they motivate this proposal. HAG-MIL in Diagnose Like a Pathologist passes only top-$k$ tiles down a quadtree from low to high magnification. PathCTM in Thinking in Scales prunes by low-resolution attention and stops early on an entropy threshold, processing about 4% of patches. AdaptivePath in Agentic Visual Reasoning in Whole-Slide Pathology Images via Active Perception trains a reinforcement-learning navigator on an "abnormality" signal. All three choose regions by how salient they are at coarse scale, and the PathCTM authors acknowledge that lesions without low-resolution salience are discarded permanently. We keep their pipelines and change only the rule that picks tiles.
Two lines of work supply the method. Online dictionary learning for sparse coding shows that a dictionary can be fit to millions of patches using only running matrices $A_t=\sum\alpha_i\alpha_i^\top$ and $B_t=\sum x_i\alpha_i^\top$, independent of stream length. We extend those statistics to also track residual energy. Dual and coupled dictionaries for super-resolution supply the idea of linking a low-resolution dictionary to a high-resolution one through shared sparse codes, and of reserving capacity for residual high-frequency detail. Examples are Image super-resolution via dual-dictionary learning and sparse representation, Coupled dictionary learning for multimodal image super-resolution and Joint Dictionary Learning for Example-based Image Super-resolution. Those works are batch-trained (K-SVD in Zhang et al.), operate on pixels, and are judged by PSNR on a handful of natural images. We never reconstruct pixels. We use the coupled model only to estimate where reconstruction would fail, and we act on that estimate.
Two other papers constrain the design. Resolution-Based Distillation for Efficient Histology Image Classification shows that low-magnification students can approach high-magnification teachers on average. This warns that most tissue is not ambiguous, so any gain must be concentrated in rare regions and will be invisible in slide-level averages unless we stratify by lesion size. Cello shows that coarse foundation-model embeddings miss micro-lesions: its 72.5% top-10% hit rate on stomach slides, against under 30% for baselines, comes from cell-level granularity, but it requires paired immunofluorescence. Our approach needs only standard H&E stains.
Research Program
Each low-magnification tile (2.5× or 5×) is embedded by a frozen pathology patch encoder into a vector $x$. Its high-magnification children (20×) are embedded and pooled into $y$. We learn a low-resolution dictionary $D_\ell$ and a high-resolution dictionary $D_h$ that share sparse codes $\alpha$, where $\alpha$ is computed from $x$ alone with an Elastic Net (an $\ell_1+\ell_2$-penalized regression). For each atom $j$ we also track how much high-resolution detail the coupled model fails to explain when that atom is active:
$$R_j=\frac{\sum_i \alpha_{ij}^2\,\|P(y_i-D_h\alpha_i)\|^2}{\sum_i \alpha_{ij}^2},\qquad s(x)=\sum_j \alpha_j(x)^2\,R_j .$$
Here $i$ ranges over tiles that have been zoomed into so far. $P$ projects onto a task-relevant subspace of the high-resolution embedding. $s(x)$ is the predicted downsampling residual of an unvisited tile with code $\alpha(x)$. Both the numerator and denominator of $R_j$ are running sums, so memory stays at $O(k^2+mk)$ as in Mairal et al.
We build $P$ in the spirit of Task-Driven Dictionary Learning but more simply. It is the span of the top discriminant directions of the slide classifier in high-resolution embedding space, so that residual stain or texture variation irrelevant to diagnosis does not inflate $s$. We will also test the full implicit-gradient task-driven version, but only as an ablation.
The dictionaries are warm-started on the training cohort and then adapted online within each test slide. This lets $R_j$ reflect that slide's own staining and tissue mix. The zoom budget $K$ is split. A fraction $1-\rho$ goes to the host model's top-attention tiles, and the fraction $\rho$ goes to the highest-$s$ tiles among the rest. A small initial random sample of zooms seeds the per-slide statistics. Every zoom then both feeds the classifier and updates $D_h$ and $R$. The deliverable is a drop-in selection module for the HAG-MIL codebase (BearCleverProud/HAG-MIL) and a reimplemented PathCTM-style top-$K$ pipeline. We will also release an analysis of when the ambiguity score and attention disagree.
Baselines
The naive reference point is exhaustive high-magnification multiple-instance learning (MIL), where every tile is embedded and aggregated by an attention model. We use CLAM, run on the same encoder features. It is not a competitor on cost; it marks the accuracy ceiling for this encoder. Attention-based Multiple Instance Learning with Mixed Supervision on the Camelyon16 Dataset reports CLAM at AUC 0.895 and mean Dice 0.215 on Camelyon16. We rerun it rather than copy those figures.
The strongest existing method is HAG-MIL, which reports AUC 0.946 and F1 0.874 on Camelyon16 in Diagnose Like a Pathologist. It is the closest competitor because it is exactly an attention-driven coarse-to-fine selector. On BRACS-3, PathCTM reports AUC 93.1% against 92.0% for exhaustive CLAM while using about 4% of patches (Thinking in Scales). That makes it the reference for the budgeted regime. We run both from the released or reimplemented code, using the same frozen encoder, the same patch budgets $K$, the same splits and the same hyperparameter search budget. Our module changes only which tiles are selected.
The ablations isolate the key ingredient. In the first, the $\rho$ share is filled with random tiles instead of high-$s$ tiles. In the second, $s$ is replaced by plain low-resolution reconstruction error under $D_\ell$, which turns it into low-resolution anomaly detection and removes the cross-resolution part. In the third, per-slide online adaptation is switched off, leaving only the frozen cohort dictionary. In the fourth, $P$ is removed. We do not expect slide-level AUC to separate methods much, since HAG-MIL is already near CLAM's mixed-supervision level (0.949). "Beating" therefore means higher recall of annotated micrometastases at equal patch budget, with slide-level AUC no lower than HAG-MIL's. On Lacuna we found no reported result for lesion-size-stratified tile recall on Camelyon16 for any of these methods, so we will measure it for all of them.
Why not just…?
Why not just mix random exploration into the attention selector? This is the most serious threat. If missed lesions are scattered with no relation to tissue type, random tiles recover them as well as any score can. The random-fill ablation tests this directly at matched budget. The proposal earns its place only if high-$s$ selection recovers significantly more micrometastases than random selection at the same $\rho$. If it does not, the conclusion is that downsampling loss is not predictable from low-resolution appearance, which would itself be a useful negative result.
Why not just raise $k$, or lower the attention threshold? Taking more of the same ranking selects more of what attention already favors. A lesion ranked at the bottom by attention stays at the bottom. PathCTM's own efficiency result says the useful information is concentrated, so the question is which tiles to add, not how many. We plot micrometastasis recall against budget for both attention-only and mixed selection. If attention-only catches up at small extra budget, our claim fails.
Why not just train a neural network offline to predict the high-resolution embedding (and its error) from the low-resolution one? This is a real alternative, and our design must justify its online, dictionary-based form against it. We will train a heteroscedastic regressor, meaning one that predicts both the high-resolution embedding and its own error, on the same training tiles, and use its predicted error as $s$. The dictionary version has two possible advantages. It adapts to each slide's stain and tissue mix from a few zooms, at constant memory and in closed form, and its per-atom residuals are inspectable. The frozen-dictionary ablation compared with the regressor shows whether per-slide adaptation is what matters. If the regressor matches the fully adaptive dictionary, we will report that a learned predictor suffices.
Why not just use a stronger encoder, or cell-level features as in Cello? The Cello evidence above suggests coarse foundation-model embeddings remain diluted. More to the point, our module sits on top of whatever encoder is used, so a better encoder raises every row of the comparison rather than replacing the selection problem.
Proposed Experiments
The primary benchmark is Camelyon16 (270 training and 129 test slides, with pixel-level metastasis masks). We stratify metastases by area into micro and macro using the annotations. We report tile-level recall of metastatic tissue at budgets of 2%, 4% and 8% of tiles, slide-level AUC and F1, and patches processed per slide. Secondary benchmarks are BRACS-3, to match PathCTM's budget regime, and TCGA-NSCLC and TCGA-RCC, to check that slide-level accuracy on subtyping does not degrade when the lesion-finding advantage is irrelevant.
The first decisive check is diagnostic. On Camelyon16 test slides, take micrometastatic tiles that attention-only HAG-MIL never selects, and ask whether $s$ ranks them in its top decile far more often than chance and than plain low-resolution anomaly scoring does. We will also report the rank correlation between $s$ and attention, which should be low. The second check is the matched-budget recall comparison against random fill and the learned regressor. The third check examines what the high-residual atoms look like. If $R_j$ is dominated by stain or focus artifacts rather than cellular morphology, even with $P$ applied, the method is measuring image quality, not diagnostic ambiguity. We will inspect the top-residual atoms' nearest tiles with a pathologist and treat an artifact-dominated result as a failure of the hypothesis.
References
- Diagnose Like a Pathologist: Transformer-Enabled Hierarchical Attention-Guided Multiple Instance Learning for Whole Slide Image Classification
- Thinking in Scales: Accelerating Gigapixel Pathology Image Analysis via Adaptive Continuous Reasoning
- Agentic Visual Reasoning in Whole-Slide Pathology Images via Active Perception
- Online dictionary learning for sparse coding
- Image Super-Resolution via Dual-Dictionary Learning And Sparse Representation
- Coupled dictionary learning for multimodal image super-resolution
- Joint Dictionary Learning for Example-based Image Super-resolution
- Resolution-based distillation for efficient histology image classification
- Cello: A Universal Cell-wise Feature Aggregation framework for Reliable Pathology Images Analysis
- Task-Driven Dictionary Learning
- Attention-based Multiple Instance Learning with Mixed Supervision on the Camelyon16 Dataset