Many imaging systems cannot see a wide area and fine detail at the same time. Gigapixel crowd video, whole-slide pathology scans, and wide-field retinal angiograms all force a choice between them. The standard software workaround is to look at a cheap low-resolution overview and then zoom in only on regions where the model is unsure. Uncertainty here usually means the model's own prediction entropy, which is high when the output probabilities are spread across classes. This saves a great deal of computation. It also fails in a predictable way. A target smaller than the overview's blur, such as a tiny tumor cluster or a pedestrian a few pixels tall, does not make the model unsure. The model simply and confidently calls it background. Entropy stays near zero, the region is never inspected, and the miss is permanent. The authors of the strongest pyramid method say so themselves (Thinking in Scales / PathCTM).
We propose two changes. First, replace "where is the model unsure?" with "where would looking closer change the answer?" A small scoring head is trained to predict, from the overview and its surrounding context, how much the task loss would drop if a given region were inspected at the next resolution level. The training targets come for free, because training images already exist at every resolution. Second, the threshold on this score is not a hand-tuned Top-K. It is calibrated on held-out data so that the rate of missed decision-relevant regions stays below a user-chosen level with stated confidence. The result is a zoom policy that comes with an explicit guarantee on what it misses, which current uncertainty-guided systems lack.
Core Claim
Low-resolution entropy measures the wrong quantity for deciding where to spend high-resolution budget. The quantity that matters is the expected reduction in loss from zooming. Call it the zoom gain of a region $p$:
$$g(p) = \ell(\hat{y}_{\text{low}}) - \ell(\hat{y}_{\text{low}+p}),$$
where $\ell$ is the task loss, $\hat{y}_{\text{low}}$ is the prediction from the overview alone, and $\hat{y}_{\text{low}+p}$ is the prediction after region $p$ is also processed at the next magnification. Unlike entropy, $g$ can be large exactly where the overview is confidently wrong.
We train a lightweight predictor $\hat{g}$ to regress this target. It sees the overview features of $p$ and of a ring of neighboring regions, so it can learn context priors ("small objects sit along walkways", "micro-foci sit near certain tissue types") as well as faint low-resolution textures that the main classifier ignores.
We then pick a threshold $\lambda$ on a calibration split. Regions with $\hat{g}(p) < \lambda$ are skipped. Let $D$ be the set of decision-relevant regions, defined for each task below. The threshold is set so that the fraction of $D$ that gets skipped stays below a target $\delta$ with probability at least $1-\alpha$, using a binomial upper confidence bound on calibration data. The claim is that, at an equal high-resolution budget, this policy recovers substantially more small or low-visibility targets than entropy- or attention-based selection. It should do so without losing overall accuracy and while actually meeting its stated miss rate on held-out data.
Relation to Prior Work
The closest work comes from two groups.
The first group motivates the gap. PathCTM walks the 5×–40× pathology pyramid. It uses low-magnification attention to load the Top-K patches at higher magnification and stops early when slide-level entropy is low. It processes about 4% of patches. The authors flag irreversible pruning of imperceptible cues as an open problem. PANDA uses an uncertainty-triggered zoom on gigapixel video for group detection. Its own detector baseline collapses on small people, which shows exactly the regime where entropy has nothing to say. We keep their coarse-to-fine structure but change the selection signal and add a miss-rate guarantee.
The second group supplies partial mechanisms. RUE separates prediction entropy from a "resolution bias" term. It models that term as a feature residual from downsampling and uses it to choose Top-K high-resolution patches for segmentation. This is the nearest idea to ours. It differs in three ways: it operates at one downsampling level, it uses a fixed K, and its residual measures lost high-frequency content rather than whether that content would change the decision. We test RUE's residual directly as a competitor scorer. The wide-area galvano-mirror system steers a high-resolution camera using a scene-context probability map plus detector variance with hand-set parameters. Its use of context priors supports our choice to give $\hat{g}$ neighborhood context. Its hand-tuned search range is what our calibrated threshold replaces. Adaptive Fourier ptychography faces the same too-high versus too-low threshold dilemma in optical acquisition. We stay in software on stored pyramids, which allow exact simulation of "what if we had zoomed here". The result would transfer to active sensors only later.
Research Program
The work has three parts.
The first part is label generation. On training images we compute $g(p)$ by running the task model with and without each region at the next level. Running a full model per region is expensive, so we use a cheap surrogate. On PANDA, $g(p)$ is approximated by the detection loss on objects whose boxes fall in $p$. In pathology, $g(p)$ is the change in slide-level loss when patch $p$'s higher-magnification embedding replaces its low-magnification one in the bag of patches the classifier pools over. Both surrogates are checked against exact recomputation on a subset.
The second part is the scorer. It is a small convolutional or attention head over frozen overview features with a context window. It is trained with a ranking loss, since only the ordering matters for selection, plus a regression term. We study how context size and the choice of overview backbone affect it.
The third part is calibration and recall. We need a fixed definition of the decision-relevant set $D$. On PANDA, $D$ is the set of cells containing a ground-truth person box that the full-resolution detector finds but the overview detector misses. In pathology, $D$ is the set of patches carrying the top 5% of attention mass in an exhaustive 40× model. This is an operational stand-in, because the file gives us no region-level lesion annotations for BRACS. We calibrate $\lambda$ on held-out slides or frames and report whether the realized miss rate on the test set falls below $\delta$.
For pyramids with more than two levels, the policy is applied at each transition. One question the program explicitly answers is whether a region skipped at 5× should be allowed back at 10× through its neighbors' context. This is the "dynamic recall" PathCTM asks for.
Baselines
The do-nothing reference is the overview alone with no zoom. Its counterpart is exhaustive full-resolution processing, which sets the accuracy ceiling and the compute cost to beat. On BRACS-3, exhaustive CLAM reports 92.0% AUC (PathCTM). On PANDA, a full-image Faster R-CNN reports AP of about 0.75 on large and about 0.20 on small people (PANDA).
The strongest existing pyramid method is PathCTM. It reports 93.1% AUC on BRACS-3 while processing about 4% of patches. We rerun it with its published selection rule and use our scorer as a drop-in replacement, so the backbone, cross-scale fusion, and early stopping stay identical. Beating it means matching or exceeding its AUC at the same patch budget while recovering a clearly larger share of $D$, the attention-heavy 40× patches.
For single-level ultra-high-resolution segmentation, the strongest competitor is RUE. It reports 79.0% mIoU on Inria Aerial and about 57 FPS at K = 50% on DeepGlobe. We replace its Top-K score with our zoom-gain score at matched K. Beating it means higher mIoU at equal K, or equal mIoU at smaller K. On PANDA small-object recall under a fixed zoom budget, we found no reported result on Lacuna, so all PANDA selection baselines are run by us.
Every scorer gets the same frozen backbone, the same high-resolution budget measured in loaded pixels, and the same hyperparameter search budget. The scorers we compare are entropy, attention, RUE's residual, the context-prior map from Long et al., and random.
Two ablations isolate the key ingredients. The first keeps the calibration procedure but swaps the scorer for entropy. If that matches our method, the gain comes from calibration, not the zoom-gain target. The second keeps the zoom-gain scorer but drops its context window. If that matches our method, context priors are not doing the work.
Why not just…?
Why not just reserve part of the budget for random exploration? This is the most serious threat. Spending, say, 10% of the high-resolution budget on uniformly random regions does put a floor under recall. That floor is roughly proportional to the fraction explored, so for rare targets it buys little per pixel. PANDA's random-crop zoom underperformed uncertainty-guided zoom (PANDA), and Long et al. found random particle search far worse than probability-guided search. Neither result tested entropy plus random exploration against a learned gain predictor, however. We therefore run entropy-plus-ε-random at several values of ε as a first-class baseline. If it matches our method on recall of $D$ at equal budget, we will report that the learned target is unnecessary and that only the calibration contribution stands.
Why not just raise K? Raising K moves along the same accuracy–cost curve without changing which regions are ranked first. Targets that entropy ranks near the bottom are recovered only once K approaches exhaustive processing, which costs about 22 times more in pathology (PathCTM). We plot recall of $D$ against budget for every scorer. The claim holds only if our curve dominates the entropy curve, not just if it beats entropy at one K.
Why not just use RUE's resolution-bias residual, which already targets downsampling loss? RUE flags regions that lost high-frequency content. Busy roof textures and road ripples lose a lot of such content while changing no decision. A three-pixel person loses little energy yet changes the outcome. That is the hypothesis, and the RUE-residual scorer in our comparison tests it directly.
Why not just tile everything at a medium resolution? Medium-resolution tiling spends compute uniformly. It still blurs the smallest targets, which is the cause of the 0.20 small-object AP on PANDA. We include it as a matched-compute reference.
Proposed Experiments
The primary testbed is PANDA. Here the targets that are invisible in the overview are real people with ground-truth boxes, so recall of $D$ is unambiguous. Overviews are downsampled 8× and 16×, zoomed regions are processed at native resolution, and budgets range from 2% to 30% of the pixels. Metrics are small-object recall and AP at each budget, realized versus target miss rate across $\delta \in \{0.05, 0.1, 0.2\}$, and wall-clock time including the scorer's overhead.
The second testbed is BRACS-3 and BRACS-7, plus TCGA-RCC, with the PathCTM pipeline at matched patch counts. Metrics are AUC, recall of the exhaustive model's high-attention 40× patches, and whether calibration holds across slides from different sources.
The third testbed is Inria Aerial and DeepGlobe, with RUE at matched K, to show the scorer transfers to dense segmentation.
The decisive check has two conditions. On PANDA and BRACS, at a matched budget, the zoom-gain scorer must recover clearly more of $D$ than entropy, attention, the RUE residual, and entropy-plus-random. In addition, the calibrated threshold must keep the test miss rate below $\delta$ in at least $1-\alpha$ of resampled test splits, while an uncalibrated Top-K violates the target at some budgets. If the scorer wins but calibration fails under cross-source shift, as can happen with slides from different sites, the guarantee is weaker than claimed. In that case we would report per-source calibration as a requirement rather than an optional extra.
References
- Thinking in Scales: Accelerating Gigapixel Pathology Image Analysis via Adaptive Continuous Reasoning
- PANDA: A Gigapixel-Level Human-Centric Video Dataset
- Boosting the Dual-Stream Architecture in Ultra-High Resolution Segmentation with Resolution-Biased Uncertainty Estimation
- A Novel Wide-Area Multiobject Detection System with High-Probability Region Searching
- Content adaptive illumination for Fourier ptychography