Zooming Where Downsampling Could Hide Something: Coupled Dictionaries as a Triage Signal for Gigapixel Pathology

Whole-slide pathology images are too large to process at full resolution, so fast methods first look at a blurry, low-magnification version of the slide. They then "zoom in", meaning they load full-resolution tissue, only where the blurry view already looks interesting. The weakness is structural. A tiny cluster of tumor cells or a patch of subtle nuclear atypia can look like ordinary tissue at low magnification, so it is never selected for a closer look. Once the method skips it, no later stage can recover it. We propose a different rule for deciding where to zoom. Instead of asking "does this region look suspicious when blurred?", we ask "how many diagnostically different high-resolution tissues could have produced this blurred view?" Regions where downsampling has thrown away enough information to hide something get inspected, even if they look bland.

The tool that answers this question is a coupled dictionary: a learned pair of vocabularies, one describing low-resolution content and one describing high-resolution content, linked so that the same code indexes matching entries in both. Classical coupled dictionaries assume one low-resolution pattern maps to one high-resolution pattern. We deliberately break that assumption and learn a one-to-many coupling. The spread of high-resolution candidates attached to each low-resolution observation then becomes an explicit, unsupervised measure of how much information downsampling destroyed at that location. This measure is computed from the slide pyramid alone, needs no labels, and comes with a readable explanation: it lists the high-resolution tissue patterns that might be hiding in the region.

Core Claim

The ambiguity that coupled-dictionary super-resolution treats as a nuisance is the right signal for allocating a zoom budget. Blind Image Super-resolution with Rich Texture-Aware Codebooks names this the "many-to-one trap", where distinct high-resolution textures collapse into the same low-resolution patch. That paper tries to escape the trap so it can output one sharp image. We want to measure the trap. Our claim is this: if a scale-space pathology pipeline adds a dictionary-derived "hidden-detail score" to its low-magnification attention, it will retrieve more of the full-resolution evidence that an exhaustive model relies on, at the same patch budget. The gain should be largest for small, low-contrast findings. Recovering pixels is not the goal, so the method never needs to produce a super-resolved image.

Background and Positioning

Two kinds of prior work motivate the problem. The first is Thinking in Scales: Accelerating Gigapixel Pathology Image Analysis via Adaptive Continuous Reasoning, which introduces PathCTM. PathCTM uses low-magnification attention as a proxy for information density, prunes background tissue, and zooms into only the top-scoring regions. It processes about 4% of patches and reaches 93.1% AUC on BRACS-3. The same work acknowledges that cues invisible at 5× are pruned irreversibly. The second is Agentic Visual Reasoning in Whole-Slide Pathology Images via Active Perception, which trains a reinforcement-learning navigator to choose zoom regions. That paper reports that pathologists who saw only the navigator's selected views reached 82.9% accuracy, against 89.7% with full slides. We read that gap as evidence that saliency-driven selection misses diagnostic content.

Both systems choose regions by how salient they look at coarse scale. We keep their pipelines and change only the selection rule.

Three lines of work supply the method:

One prior result constrains the design. Resolution-Based Distillation for Efficient Histology Image Classification shows that a low-resolution student can approach a high-resolution teacher at moderate downsampling, but degrades at 0.625× unless given large amounts of unlabeled data. This tells us that low-resolution features alone carry less and less diagnostic content as scale shrinks. Our approach therefore does not try to make the low-resolution view sufficient. It only tries to tell us where it is insufficient.

Method

We work in the embedding space of a frozen pathology patch encoder, not in pixel space. This keeps sparse coding cheap, avoiding the tens-of-minutes latency reported for pixel-level coupled sparse coding in Unsupervised Cross-Modal Synthesis of Subject-Specific Scans. Training pairs come for free from the slide pyramid. For each location, $x$ is the embedding of a 5× tile and $y$ is the pooled embedding of its corresponding 20× sub-tiles.

Each of the $K$ low-resolution atoms (the columns of $D_{LR}$) is linked to $M$ high-resolution atoms, which form a block $D_{HR}^{(k)}$, with learned mixing weights $\pi_{km}$. Training minimizes

$$\sum_i \|x_i - D_{LR}\alpha_i\|_2^2 + \min_{m}\Big\|y_i - \sum_k \alpha_{ik} D_{HR}^{(k)}[:,m]\Big\|_2^2 + \lambda\|\alpha_i\|_1 ,$$

where $\alpha_i$ is the shared sparse code. The inner minimum over $m$ is relaxed to a soft assignment during training, which lets several distinct high-resolution atoms explain the same low-resolution atom.

At inference, a low-resolution tile is sparse-coded once. Its hidden-detail score is

$$h(x) = \mathrm{Var}_{m \sim \pi}\big[\, f(\hat y_m) \,\big],$$

where $\hat y_m$ ranges over the high-resolution candidate embeddings implied by the code, and $f$ is the downstream attention or diagnostic head. The score is high only when the candidates disagree in a way the diagnosis cares about. Smooth regions with diverse but diagnostically irrelevant texture do not trigger zooms.

The zoom budget is then split between PathCTM's attention ranking and a ranking by $h$. The split is a single mixing parameter, tuned on validation slides. Everything after selection stays exactly as in PathCTM. The only learned component that touches labels is the head $f$, which already exists, so the dictionary itself is trained without supervision.

Research Program

The work has three stages.

Stage 1: test the coupling itself. Before any downstream use, we check whether a one-to-many coupling on pyramid pairs places high $h$ where the full-resolution embedding actually changes the diagnostic head's output. We also check that $\pi$ does not collapse onto a single candidate per atom. We track atom utilization because collapse is a documented failure: Blind Image Super-resolution with Rich Texture-Aware Codebooks reports 3.2% codebook utilization for FeMaSR against about 77% for its own model.

Stage 2: plug the score into a pruning pipeline. We add $h$ to PathCTM-style pruning and measure retrieval and slide-level accuracy across budgets.

Stage 3: inspect the candidates. Following the HOGgles idea, we retrieve the training tiles nearest to the competing high-resolution candidates. Pathologists then judge whether the "possibly hidden" patterns are morphologically meaningful, for example atypia versus benign stroma, rather than staining noise.

Baselines

Do-nothing reference. A model trained on downsampled 5× tiles only, with no zooming. This measures how much diagnostic signal survives downsampling unaided.

Upper reference. Exhaustive native-resolution multiple instance learning, meaning a slide classifier that aggregates every full-resolution patch, using CLAM. Thinking in Scales reports 92.0% AUC for CLAM on BRACS-3. We also use this model to produce the reference "important patches", defined as its top-attended 20× patches.

Strongest existing method. PathCTM, at its reported 93.1% AUC on BRACS-3. We re-run it with the same frozen encoder, splits, and patch budget as our method. Beating it means two things at matched budget: higher recall of the exhaustive model's top patches, and equal or better AUC. For a claim of robustness to small findings, we additionally require a recall gain concentrated on patches whose attention rank at 5× is low.

Calibrated-uncertainty competitor. Ranks regions by Monte-Carlo dropout variance of the low-resolution attention head.

Learned-ambiguity competitor. A mixture-density network, a regressor that outputs a distribution rather than a single value, trained on the same pyramid pairs to predict the high-resolution embedding from the low-resolution one. It is given the same parameter budget and tuning trials as the dictionary.

Ablations. Two ablations isolate the key ingredient:

  • One-to-one coupled dictionary. This model has no candidate spread, so its only available score is reconstruction residual.
  • Uncoupled low-resolution dictionary. This scores regions by how poorly the low-resolution vocabulary explains them, which measures novelty rather than hidden detail.

Why not just…?

Why not just lower PathCTM's pruning threshold? A lower threshold buys recall with budget, and the extra budget follows the same coarse saliency ranking. It therefore still spends patches on tissue that already looks suspicious and keeps skipping bland-looking regions. We test this directly with recall-versus-budget curves. If a uniformly larger budget matches our curve, the score adds nothing.

Why not just distill a high-resolution teacher into a low-resolution student? Resolution-Based Distillation shows this works at 5× but loses accuracy at 0.625×. More importantly, a student outputs a prediction, not a location. It cannot tell a triage pipeline where its own view is inadequate. We include the distilled student as a low-resolution-only reference to show where it falls short.

Why not just learn the ambiguity with a neural density model? This is the real threat. A mixture-density network trained on the same pairs could learn the same spread. Our hypothesis is that sparse coupling gives better-calibrated spread in rare morphologies, because rare patterns get their own atoms instead of being averaged away, and that it gives inspectable candidates. If the mixture-density baseline matches us on recall at every budget, we would report the dictionary as an interpretability layer rather than a retrieval improvement.

Why not just use predictive uncertainty of the coarse model? Dropout variance measures the model's uncertainty about its own output on the observed blurred tile. It does not measure uncertainty about content that downsampling removed. A tile that is confidently "benign stroma" at 5× can still be ambiguous at 20×. The comparison is run under the same protocol.

Proposed Experiments

Primary benchmark: BRACS. We use BRACS-3 and the harder 7-class BRACS-7 task, where fine distinctions should make hidden detail matter most. We report AUC at matched patch budgets (2%, 4%, 8%) and recall of the exhaustive model's top-100 patches. No reported BRACS-7 number was found on Lacuna for PathCTM, so we will establish one. For TCGA-RCC staging, no reported numeric result was found on Lacuna for PathCTM either, and we will re-run it there.

Decisive check. Stratify the exhaustive model's top patches by their 5× attention rank. The hypothesis predicts that our gain over PathCTM is concentrated in the bottom-ranked stratum, where the irreversible-pruning failure lives. If gains are uniform across strata, or come only from high-saliency regions, the proposed mechanism is wrong even if AUC improves.

Secondary check: celiac and lung adenocarcinoma. If the celiac and lung adenocarcinoma cohorts from Resolution-Based Distillation are accessible, we test whether $h$ grows as magnification drops toward 0.625×, where that paper's student degrades. If it does, the score tracks real information loss.

Pathologist audit. For 100 high-$h$, low-attention regions, we ask whether the retrieved high-resolution candidates contain diagnostically different morphology. This distinguishes meaningful ambiguity from texture variance.