Magnification as Time: Zoom-Video Models for Gigapixel Pathology

A pathologist does not read a slide one magnification at a time. They sweep at low power, notice something faintly odd, and zoom in smoothly while the tissue resolves. Their sense of what they will see at high power guides where they zoom. Current computer models for whole-slide images, which are gigapixel scans of tissue, lack this sense. Efficient methods look at a cheap low-magnification view, rank regions by how salient they look there, and load high-magnification pixels only for the top regions. When a tiny metastasis is not salient at low power, it is thrown away for good. Both ZoomMIL and PathCTM name this "irreversible pruning" failure, and PathCTM leaves the fix as future work.

We propose to treat the magnification axis as the time axis of a video. For any point on a slide, we render a short "zoom clip": a sequence of crops centred on that point at steadily increasing magnification, sampled anywhere in a continuous range rather than only at the scanner's fixed levels. We then train one encoder jointly on single-magnification patches, used as one-frame videos, and on zoom clips, the way text-to-video models are trained jointly on images and videos. The video part of the training teaches the model how tissue appearance changes as you zoom, so it learns to predict what high magnification will reveal from a low-magnification view. The zoom policy then ranks regions by predicted high-power evidence instead of by low-power attention. Our claim is that this anticipatory zoom policy recovers small lesions that attention-based pruning misses, at the same compute budget.

Core Claim

Treating scale as time does three things at once. Clips are local, so they fit in GPU memory, and the patch encoder can be fine-tuned end to end rather than frozen. Fusion across scales becomes temporal modelling instead of feature concatenation. And the choice of which magnifications to look at becomes a continuous variable the model can optimize, instead of a fixed menu of 5×/10×/20×. The specific, testable prediction is narrower than all three: when deciding where to zoom, a model trained to anticipate high-magnification features through zoom-video training should miss fewer small diagnostic regions than ZoomMIL-style top-K attention routing.

Relation to Prior Work

The closest work is VINO. It also turns a slide into "synthetic video clips" to escape the memory limit and fine-tune a ResNet-18. Its clips, however, are raster sequences of neighbouring patches at a single magnification. Time there is a flattened spatial axis that breaks 2D adjacency, and there is no scale and no zoom decision. Our clips move along scale at a fixed location, which is the axis on which pathologists actually reason sequentially.

Two efficient-inference methods supply the setting we compete in. ZoomMIL learns a differentiable top-K selection that routes patches from 5× to 10× to 20× and fuses them by sum-pooling. It cuts compute 40–50×, but its authors report false negatives on micrometastases in CAMELYON16. PathCTM adds iterative refinement and entropy-based early stopping over a 5×–40× pyramid, and it also prunes by low-resolution attention. Both decide to zoom from what is visible now. We decide from what is predicted to become visible.

Other multi-scale pathology models fix their scales in advance. MS-DA-MIL concatenates 10× and 20× embeddings. PTree-Net links 1.25×/5×/10× patches in a parent–child tree. TransMIL captures multi-scale spatial layout only through convolution kernel sizes. A result from DSMIL constrains our design: adding a 1.25× level hurt performance relative to a focused 5×/20× pyramid. Indiscriminate scale fusion adds noise, which is why we learn which magnifications to visit rather than fusing all of them.

The method borrows from three places. The joint-training recipe comes from Video Diffusion Models, which bypasses temporal layers on image batches, and from Snap Video, which treats an image as a video at infinite frame rate. In our setting, a single-magnification patch is a clip with zero zoom velocity. The split into a scale-invariant content vector plus per-frame residuals follows Jointly Trained Image and Video Generation using Residual Vectors. The idea of relaxing a discrete design menu into learnable continuous windows comes from DNArch. DNArch did this for kernel sizes and downsampling rates on fixed-resolution inputs. We apply it to the input magnification itself, which DNArch never addresses.

Research Program

The first component is a zoom-clip renderer. Let $s = \log_2 m$ be log-magnification, with $m$ ranging over a continuous interval such as 1.25× to 40×. For a tissue point, a clip is $T$ crops of fixed pixel size at magnifications $s_1 < \dots < s_T$. Each crop is resampled with anti-aliasing from the nearest higher pyramid level, so its field of view shrinks as $s$ grows. Training clips use randomly drawn, irregularly spaced $s_t$, so the model cannot memorize the scanner's levels.

The second component is a jointly trained encoder. It is a patch transformer with interleaved spatial and scale-temporal attention, and the temporal layers receive $s_t$ as a continuous positional input. Image batches are drawn at random single magnifications from the abundant pool of patches and skip the temporal layers. Clip batches use all layers. Besides the slide-level weak label used by an attention-based multiple-instance head (a model that pools many unlabeled patches into one slide prediction), the clip stream is trained to anticipate later frames:

$$\mathcal{L}_{\text{zoom}} = \sum_{t<T} \big\| g_\theta(h_{\le t}, s_{t+1}) - \mathrm{sg}[f_\theta(x_{t+1})] \big\|^2 .$$

Here $h_{\le t}$ is the encoder's state after the first $t$ frames, $x_{t+1}$ is the next crop, $f_\theta$ is the per-frame encoder, and $g_\theta$ predicts the embedding at a requested magnification. The operator $\mathrm{sg}$ stops gradients to prevent the target collapsing. Image batches carry a magnification-label auxiliary loss, so the spatial features stay scale-aware.

The third component is the anticipatory zoom policy. At inference, every region is encoded once at low magnification. For each region, the predictor $g_\theta$ produces embeddings at candidate higher magnifications, and the multiple-instance head scores them. The region's priority is its maximum predicted diagnostic score over the candidates. Regions are expanded in priority order until a fixed patch budget runs out. The magnification each region is expanded to is chosen by a learnable soft window over $s$, in the style of DNArch's Gaussian masks, trained with a compute penalty on expected pixels loaded. Because priority comes from a prediction, a region that looks bland at low power but whose texture forecasts atypia can still be expanded. This is the recall mechanism that ZoomMIL and PathCTM lack.

Baselines

The naive reference point is exhaustive inference. Every tissue patch is encoded at 20× by the same fine-tuned backbone and pooled with attention-based multiple-instance learning or TransMIL. This gives the accuracy ceiling for a single magnification and the compute cost to beat. For context, published numbers on CAMELYON16 include TransMIL at test AUC 93.09% and DSMIL at 89.9% accuracy with localization FROC 0.437.

The strongest existing method on accuracy is VINO, which reports bag AUC 0.9466 and instance-detection AUC 0.9213 on CAMELYON16 and bag AUC 0.9853 on TCGA-NSCLC. We will rerun it with the same backbone and splits. Matching or exceeding it on bag AUC while loading a small fraction of high-magnification patches would count as beating it.

The strongest competitor on the actual question, efficient zooming without missed lesions, is ZoomMIL. We found no reported ZoomMIL AUC on CAMELYON16 on Lacuna, only its 40–50× FLOPs reduction. We will reimplement it with our backbone and an identical patch budget. We will also reimplement PathCTM-style attention pruning; its only reported number in the file is BRACS-3 AUC 93.1% versus CLAM at 92.0% in PathCTM.

All methods get the same training slides, the same encoder initialization, the same number of tuning trials, and the same high-magnification patch budgets, swept from 2% to 20% of tissue patches. Beating ZoomMIL means higher lesion recall at equal budget, with slide AUC no lower.

Three ablations isolate the key ingredient. The first removes the video stream: an image-only encoder plus a regressor from low- to high-magnification features. The second removes anticipation and routes by low-magnification attention inside our own architecture. The third restricts magnifications to the discrete 5×/10×/20× pyramid.

Why not just…?

Why not just encode everything at 20× with a strong pathology foundation encoder? This is the accuracy ceiling, and it may well win on AUC alone. The open problem the file documents is cost: 10,000–50,000 patches per slide, most of them irrelevant. Our claim is about the accuracy-versus-budget curve, not peak AUC. If our method falls more than a point of AUC short of exhaustive inference at a 10% budget, the efficiency argument fails, and the exhaustive baseline is there to show it.

Why not just concatenate features from a few fixed magnifications, as MS-DA-MIL does? Fixed concatenation still pays for every patch at every scale. DSMIL's finding that adding 1.25× degraded performance shows that indiscriminate fusion can hurt. The discrete-pyramid ablation measures how much continuous, learned scale selection adds over a fixed menu. If the gain is negligible, we will report that continuity is unnecessary and keep only the anticipation component.

Why not just train a regressor that predicts 20× attention or features from 5× features, with no video framing? This is the real threat. It captures the anticipation idea with far less machinery. The image-only regressor ablation tests it directly with matched parameters and data. The video framing earns its place only if two things hold: training on irregularly sampled zoom clips gives better recall on small lesions than a single fixed low-to-high mapping, and it transfers to magnifications not seen during training. If the regressor matches us, the paper's contribution reduces to "anticipatory routing," and we will say so.

Why not just add a random exploration budget to ZoomMIL so that some pruned regions get revisited? Random revisits do recover some misses, but recall then scales only with the fraction of budget spent randomly. We will include ZoomMIL plus uniform random recall at equal total budget. Anticipation should beat it specifically on the smallest lesions, where random revisits rarely land.

Proposed Experiments

The decisive experiment uses CAMELYON16, whose lesion annotations let us measure what pruning methods hide. For each method and budget, we report slide AUC, FROC, and lesion recall: the fraction of annotated lesions with at least one patch examined at high magnification. Lesion recall is stratified by lesion size, so micrometastases are reported separately. The hypothesis holds if anticipatory routing raises small-lesion recall over ZoomMIL and attention pruning at equal budget, and if the image-only regressor ablation captures clearly less of that gain.

Two further tasks test generalization. BRACS-3 and BRACS-7 test fine-grained grading, where the diagnostic evidence is cellular atypia that is invisible at low power. TCGA-NSCLC tests a task where low power may already suffice; here we expect early exits and little advantage, which is an honest negative control. A held-out-magnification test trains with magnifications above 20× withheld and evaluates zoom prediction at 40×. This checks whether the scale-temporal model has learned zoom dynamics or only the training pyramid.

We will also report wall-clock time and pixels loaded per slide. We will plot the learned magnification windows to see whether the model rediscovers DSMIL's preference against very low magnification or finds intermediate scales that human menus omit.