Pathology slides are scanned at roughly 100,000 × 100,000 pixels. That is far too large for a GPU, so today's segmentation models cut the slide into thousands of tiles, label each tile, and stitch the results back together. Generative image models face a related problem. They build an image one piece at a time, and every step needs a full pass through the network. The fastest recent fix, next-scale prediction, generates a whole coarse image first and then each finer resolution in one parallel step. It removes most of the step-by-step waiting. It still pays for every token at every scale, though, and at slide resolution the finest map alone would hold tens of millions of tokens.
We propose to generate a slide's segmentation mask from coarse to fine, and at each finer scale to predict only the tokens whose label is still undecided. In practice these are tokens near tissue boundaries. Every other token copies its label from its parent at the coarser scale. The model reads high-magnification pixels from disk only for the undecided tokens. The cost then grows with the length of the object boundaries rather than with slide area, while every prediction can still attend to the full coarse context of the slide. The central risk is that coarse errors become permanent: a small structure missed at low magnification would never be revisited. So the design adds a cheap parallel check at each scale that can reopen copied tokens. This mirrors how speculative decoding in language models proposes cheap draft tokens and then verifies them.
Core Claim
A single next-scale autoregressive model, run sparsely over the physical magnification pyramid of a slide with draft-and-verify reopening, can match the slide-level Dice of the strongest end-to-end tiled segmenter. It should do so while reading a small fraction of the high-magnification pixels and using a fraction of the wall-clock time and memory.
The cost argument is simple geometry. Let $N_k$ be the number of tokens processed at scale $k$, where the token grid doubles in side length at each scale. For regions bounded by smooth curves, the undecided tokens lie in a band around the boundaries, so
$$N_k \approx c\,P\,2^{k} + \rho\,4^{k},$$
where $P$ is the total boundary length measured in coarsest-scale token units and $c$ is the band width. The second term covers reopened tokens: $\rho$ is the fraction of copied tokens the verifier reopens, and $4^k$ is the full grid size at scale $k$. Dense next-scale decoding pays $4^k$ at every scale. The sparse scheme pays roughly $2^k$ as long as $\rho$ stays small. Measuring $\rho$ and $c$ on real slides is therefore itself a result: it tells us whether pathology masks are "boundary-dominated" enough for the argument to hold.
Relation to Prior Work
Three works supply the mechanism, and each stops short of this setting.
Visual Autoregressive Modeling (VAR) introduced next-scale prediction. It generates each resolution's full token map in parallel, conditioned on all coarser maps, and reports FID 1.73 on ImageNet 256 and a 20×–45× speedup over raster-order generation. We reuse the scale-wise conditioning but drop VAR's assumption that every scale is decoded densely.
ARGenSeg showed that segmentation masks can be treated as multi-scale token maps and generated with next-scale prediction. It reaches 86.3 cIoU on RefCOCO val and runs 4×–10× faster than sequential generators. It works only on natural images up to a 16 × 16 token map. Our proposal inherits its mask-as-token-pyramid idea and moves it to a gigapixel coordinate field.
The speculative decoders for images, Annealed Relaxation of Speculative Decoding and CSD, draft several tokens and verify them in parallel. CSD reports that relaxing acceptance criteria causes texture artifacts. That negative result constrains our design: our drafts are copied parent labels, and the verification target is a discrete class label rather than texture. Aggressive acceptance should therefore be much less harmful, and we will test this directly.
On the pathology side, two works motivate the gap.
HoloHisto samples 4K patches directly from the slide, compresses them into tokens with an ImageNet-trained VQGAN, and segments with linear attention. It reaches 84.54% WSI-level Dice on the kidney glomerulus segmentation benchmark (KPIS) and 88.85% average Dice on the multi-structure kidney benchmark (NEPTUNE). It still covers the whole slide with 4K windows, and its tokens are compression features rather than a modeled distribution.
HAG-MIL zooms through a quadtree and sends compute only to the top-$k$ suspicious patches at each level. That is the closest idea to our gating. However, it is a slide-level classifier with fixed cutoffs, and it prunes irreversibly. Our proposal differs in three ways: it uses predictive uncertainty over labels, it outputs dense masks, and it allows pruned regions to be reopened.
Two further works mark the boundaries of the problem.
AdaptivePath chooses where and at what magnification to look using a learned policy. Each observation is a sequential decision, which reintroduces exactly the latency this proposal removes.
ZoomLDM generates pathology images across magnifications using diffusion, a generator that learns by gradually denoising random noise into images. It takes about 8 minutes per 4096 × 4096 image. It shows that cross-scale generative conditioning is feasible in pathology, but it does not target segmentation or speed at slide scale.
Research Program
Pyramid and tokens. The model operates on the slide's native pyramid, from about 1.25× up to 20× magnification.
- The mask at scale $k$ is represented as a grid of label tokens. For binary glomerulus masks we use a label grid with an explicit "mixed" class. For NEPTUNE's six structures we use a small multi-scale mask quantizer in the style of ARGenSeg.
- The image side at scale $k$ is a grid of patch features from a pathology encoder. These are computed densely at the two coarsest magnifications, where the whole slide fits in memory. At finer scales they are computed lazily, only for patches under open tokens, using a random-access reader of the kind HoloHisto uses.
Model. One transformer is shared across scales and carries a scale embedding. When predicting an open token, it attends to:
- the full coarse context (all tokens at the two coarsest scales),
- the ancestor chain of the token, and
- a local window of siblings.
Attention cost therefore stays bounded no matter how large the slide is.
Training. Training uses teacher forcing on ground-truth mask pyramids. Open sets are sampled so that the model sees realistic frontiers, including deliberately perturbed ones.
Inference at each scale. Each scale proceeds in three steps:
- Draft. Every child token of a decided parent copies the parent's label.
- Verify. A cheap verifier head runs in parallel on all drafted tokens. It uses only already-available coarse image features plus the cheapest newly fetched features on a thin band around the frontier. Any draft whose label probability falls below a threshold is reopened.
- Decode. All open tokens are decoded in one parallel step.
The threshold is the single knob that trades accuracy for cost. We will report the full accuracy–cost curve rather than a single point.
Analysis. Beyond accuracy, we will measure the geometric quantities $c$ and $\rho$ per organ and structure. We will also measure the miss rate for small objects, which is the failure mode the verifier exists to catch, separately by object size.
Baselines
The naive reference point is dense tiled segmentation: a standard U-Net over the whole slide at 20× with overlap-and-stitch. The file reports 62.19% Dice for U-Nets on KPIS in the HoloHisto evaluation. We will also run it ourselves to obtain its cost numbers.
The strongest existing method is HoloHisto-4K. Its reported results are 84.54% WSI-level Dice on KPIS and 88.85% average Dice on NEPTUNE, both from the HoloHisto evaluation ( HoloHisto: End-to-end Gigapixel WSI Segmentation with 4K Resolution Sequential Tokenization). SAM-ViT-H at 77.24% on KPIS is a secondary reference. We will rerun HoloHisto on the same splits and hardware to measure wall-clock time per slide, pixels read at 20×, and peak memory, none of which it reports.
Beating HoloHisto means one of two outcomes:
- equal Dice (within 1 point) at no more than a quarter of its 20× pixel reads and wall-clock time, or
- higher Dice at equal cost.
The second outcome would most plausibly come from better continuity across tile boundaries, because every token sees slide-level context.
A dense next-scale variant uses the same model with no sparsity. We run it at the 4K patch level, where it fits in memory, to isolate whether sparsity costs accuracy.
Two ablations isolate the key ingredients:
- Greedy cascade: the same model with verification disabled. Copied tokens are never reopened.
- Parent-only conditioning: each scale sees only its immediate parent, which removes cross-scale autoregressive context.
An attention-based top-$k$ gate in the style of HAG-MIL, using its public code for the quadtree machinery, replaces our uncertainty gate at a matched token budget.
All methods share the same image encoder where applicable, the same training data, the same number of GPU-hours for tuning, and the same threshold search on a validation split.
Why not just…?
Why not just run HoloHisto with its 4K windows over the whole slide? It is the accuracy leader and is simple. Its cost, however, scales with tissue area: it must read and encode every 20× pixel. For sparse structures like glomeruli, most of those pixels sit far from any boundary. If our method's pixel-read fraction at matched Dice is not well below one, this alternative wins, and the accuracy–cost curve against HoloHisto will show it plainly.
Why not just use a discriminative coarse-to-fine cascade: segment at low power, then refine uncertain boundaries with a separate network at high power? This is the real threat, because it captures most of the perimeter-scaling benefit without any autoregressive machinery. Our bet is that two ingredients matter:
- reopening, which the cascade lacks, recovers structures missed at coarse scale, and
- conditioning on all ancestors plus slide-wide coarse context, rather than on the parent alone, reduces errors on ambiguous boundaries.
The greedy-cascade and parent-only ablations test exactly this, stratified by object size. If both ablations match the full model on KPIS and NEPTUNE, we will report that the autoregressive framing adds nothing beyond uncertainty-gated refinement. That would be a useful negative result for the next-scale literature.
Why not just use MaskGIT-style parallel masked decoding? MaskGIT decodes a whole token grid in 8–12 rounds, 30×–64× faster than raster generation. Its grid is still the full finest-resolution canvas, so memory and compute remain proportional to slide area. It also reports that long-range outpainting forgets distant context. Sparsity, not the number of decoding rounds, is the binding constraint at gigapixel scale.
Why not just prune with top-$k$ attention like HAG-MIL? Top-$k$ selects regions that look suspicious. Segmentation needs regions whose fine labels are unpredictable from coarse evidence, and these differ: a large tumor interior is suspicious but perfectly predictable. The matched-budget gate comparison measures this difference directly.
Proposed Experiments
Main accuracy comparison. We use KPIS WSI-level Dice and NEPTUNE average Dice with the official splits. The headline metrics are:
- Dice,
- wall-clock seconds per slide,
- fraction of 20× pixels read, and
- peak GPU memory,
all reported as curves over the verification threshold.
Scaling test. We plot processed tokens per scale against scale index on held-out slides. The plot should show slope near 2 on the log-base-2 scale for the sparse model, meaning tokens double per scale, versus 4 for dense decoding. We will also report the measured $c$ and $\rho$ for each structure type, since NEPTUNE's thin structures may be less boundary-dominated than glomeruli.
Small-object recall. We report recall for objects under a size threshold, with and without verification. The claim fails if verification does not recover most of the recall lost by the greedy cascade.
Tokenizer check. For NEPTUNE's quantized masks and for image features, we compare an ImageNet-trained quantizer against one trained on pathology tiles. This addresses the domain mismatch HoloHisto reports.
We are not aware of any reported latency or pixel-read numbers for HoloHisto; no reported result found on Lacuna for per-slide runtime on KPIS. We will therefore measure these quantities ourselves under matched hardware.
References
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
- ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
- Annealed Relaxation of Speculative Decoding for Faster Autoregressive Image Generation
- CSD: Content-aware Speculative Decoding for Efficient Image Generation
- HoloHisto: End-to-end Gigapixel WSI Segmentation with 4K Resolution Sequential Tokenization
- Diagnose Like a Pathologist: Transformer-Enabled Hierarchical Attention-Guided Multiple Instance Learning for Whole Slide Image Classification
- Agentic Visual Reasoning in Whole-Slide Pathology Images via Active Perception
- ZoomLDM: Latent Diffusion Model for multi-scale image generation
- HoloHisto: End-to-end Gigapixel WSI Segmentation with 4K Resolution Sequential Tokenization
- MaskGIT: Masked Generative Image Transformer