Whole-slide pathology images are often 100,000 pixels on a side. No GPU can process one in a single pass, so nearly every method cuts the slide into thousands of small tiles. Each tile goes through a frozen pretrained encoder once, and a lightweight "aggregator" learns to combine the resulting tile vectors into a slide diagnosis. Recent aggregators such as Transformers, state-space scanners and graph networks have become good at relating tiles to one another. Information still flows only upward, though. The encoder that looks at a tile never learns what the rest of the slide contains, so a tile at an invasive tumor front is encoded exactly as it would be on a benign slide. We propose to send slide context back down. A cheap first pass over cached tile vectors builds a coarse two-dimensional "context map" aligned with slide coordinates. A second pass re-encodes selected tiles with that map controlling per-location scale-and-shift operations on the encoder's internal feature maps. These operations are spatial feature transform layers, which multiply and offset intermediate activations by location-dependent amounts.
The opportunity matters because the reported failures of current systems are failures of what tiles reveal, not of how they are combined. Attention maps from weakly supervised models are noisy enough that tile annotations raise localization Dice substantially (Attention-based MIL with Mixed Supervision). Patch classifiers also flip their predictions many times as surrounding context changes (Does context matter in digital pathology?). If context can change what features the encoder extracts, and not only how much each tile counts, these two failures should shrink. We give a concrete test that would show if it does not.
Core Claim
Context-conditioned spatial modulation of a frozen patch encoder's intermediate layers should improve tumor localization on weakly labeled slides. The gain should exceed what is possible from (a) contextualizing the already-computed tile embeddings and (b) context-free parameter-efficient adaptation of the encoder. It should also make tile-level predictions more stable when the visible context changes. The mechanism at encoder layer $l$ is
$$F'_l(u) = \big(1 + \gamma_l(C(u))\big) \odot F_l(u) + \beta_l(C(u)),$$
where $F_l(u)$ is the encoder's token feature at slide coordinate $u$. $C$ is the context map, defined on the grid of tile positions and bilinearly interpolated to the sub-tile token grid. $\gamma_l$ and $\beta_l$ are small learned networks initialized to output zero, so training starts from the frozen encoder unchanged. Because $C$ is interpolated between neighboring tiles, a tile straddling a tumor margin receives different modulation on its two sides. The modulation is therefore genuinely spatial, not a single per-tile vector.
The closest prior work does parts of this, and the contrasts are the substance of the claim. Spatial feature transforms come from image restoration. FxSR modulates super-resolution features with a user-supplied 2D control map. We reuse that layer but infer the map from the image's own global context. StyleMapGAN shows that a 2D latent grid supports local control. We borrow the idea of a coordinate-aligned conditioning grid. Among slide models, HookMIL comes nearest: learnable anchor tokens gather slide context and feed it back to instance tokens. That feedback acts on finished 1D embeddings, after the encoder has discarded whatever it did not deem relevant. TransMIL, MambaMIL+ and the graph model of Adnan et al. relate tiles to each other but never touch the encoder. PathFiT does touch the encoder, adapting pathology foundation models with low-rank updates, but its update is the same for every tile of every slide. Ours varies with slide and location.
Research Program
The system runs in two passes per slide. In the first pass, tiles are encoded once by a frozen pathology foundation model (UNI, as used in PathFiT) and cached, exactly as in standard pipelines. A context network then operates on these cached vectors laid out on their 2D slide grid. It is a few depth-wise convolutions over the grid, in the spirit of TransMIL's position-encoding convolutions, plus a small set of HookMIL-style global anchor tokens. Keeping the tiles on a 2D grid, rather than flattening them into a scan sequence, avoids the directional bias that MambaMIL+ has to patch over. The output is the context map $C$, one vector per tile position.
In the second pass, a subset of $K$ tiles is re-encoded with the modulation layers active. These are the tiles the aggregator currently attends to most, plus random tiles. Their modulated embeddings replace the cached ones in the bag, and an attention aggregator produces the slide prediction. Gradients flow through only $K$ encoder passes. They reach the modulation networks, the context network and the aggregator, while the foundation model weights stay frozen. This keeps memory near that of fine-tuning on $K$ tiles; $K=64$ is the starting point. The design deliberately sidesteps the frozen-encoder bottleneck identified in M2Fusion and the MIL literature without requiring end-to-end gigapixel backpropagation.
At inference, all tiles are re-encoded, which roughly doubles encoder cost. We will report this cost honestly. Training on $K$ modulated tiles but testing on all of them creates a mismatch. We will measure its effect by evaluating with $K$, $4K$ and all tiles modulated, and if needed counter it with a schedule that grows $K$. A second feedback round, in which the context is recomputed from modulated embeddings, is an optional extension we will test only if one round helps.
Three analyses accompany the build. First, a probing study asks whether modulated embeddings linearly encode tile-level tumor labels that frozen embeddings do not, using Camelyon16 pixel annotations. This is the most direct test that modulation changes what is extracted. Second, a slide-level version of the "swinging" test extends Tomaszewska et al.'s patch-scale protocol to whole slides. For annotated tumor regions, we vary how much surrounding slide is visible to the context network and count tile-prediction flips. Third, the learned $\gamma$ and $\beta$ fields are visualized against annotations to see whether modulation concentrates at tumor boundaries or diffuses everywhere.
Baselines
The naive reference point is attention-based multiple instance learning (pooling tiles with learned scalar weights) on the same frozen UNI tiles with no context. For this we use CLAM as in Tourniaire et al., who report Camelyon16 slide AUC 0.895 and localization Dice 0.215 for plain CLAM. Their numbers used a different encoder, so we rerun CLAM on our features. Anything that fails to beat this rerun is uninteresting.
The strongest existing context aggregators are TransMIL, HookMIL and DSMIL. TransMIL reports 93.09% test AUC on Camelyon16, 96.03% on TCGA-NSCLC and 98.82% on TCGA-RCC. HookMIL reports 94.42% AUC on PANDA and 66.42% on BCNB-HER2. DSMIL reports 89.9% accuracy and a localization FROC (a detection score averaged over false-positive rates) of 0.437 on Camelyon16. For MambaMIL+, no reported per-dataset result was found on Lacuna, only average relative gains. We will rerun all four on identical UNI features, splits and hyperparameter search budgets. We report each paper's own number alongside, to flag any gap between our reproduction and the original.
HookMIL is the most important competitor because it already feeds context back to instances. Beating it on slide AUC alone would not be convincing, since Camelyon16 AUC is already above 93%. The bar is a gain in localization Dice and FROC, with slide AUC no worse.
Two ablations isolate the key ingredient. The embedding-level feedback ablation feeds the identical context map into a scale-and-shift on the final cached embedding instead of inside the encoder. It uses matched parameter count and identical training, so the only difference is where the modulation acts. The context-free adaptation ablation replaces context-conditioned modulation with PathFiT-style low-rank encoder adaptation at matched trainable parameters and the same $K$-tile training. It tests whether gains come from adapting the encoder at all rather than from slide context.
Three controls follow. A shuffled-context control conditions each slide on another slide's context map. A self-conditioning control conditions each tile only on its own frozen embedding. A per-tile constant control replaces the interpolated map with one vector per tile, which isolates sub-tile spatial variation. All methods get the same GPU-hour budget and the same number of tuning trials.
Why not just…?
Why not just contextualize the cached embeddings, as HookMIL does? This is the real threat. Embedding-level feedback is cheaper and may capture most of the benefit. Our argument is that a frozen encoder's output has already discarded information, such as fine nuclear detail it judged irrelevant, and no operation on that output can recover it. That argument is only a hypothesis. The embedding-level ablation and the linear probe settle it. If modulated embeddings do not encode tile labels better than frozen ones, and the ablation matches the full method on Dice and FROC, we will conclude that internal modulation is unnecessary and report that.
Why not just fine-tune the encoder with LoRA, as PathFiT does? PathFiT shows adaptation helps, for example a 10.20% gain on non-H&E tasks. Its adapters are identical for every tile, so they cannot make a tile's encoding depend on its neighborhood. The matched-parameter context-free ablation measures exactly the part of our gain attributable to context. The shuffled-context control should fall to that level if context is doing the work.
Why not just use larger tiles, as HoloHisto does? HoloHisto trains end-to-end on 4K windows and reaches 84.54% whole-slide Dice on KPIS. Even 4K windows are a small fraction of a slide, and memory grows with the window. Our approach gives every tile slide-wide context at the cost of a second encoder pass. This is complementary rather than competing. A larger-tile CLAM baseline at matched compute will show whether a wider field of view alone accounts for the localization gains.
Why not just add tile-level annotations? Mixed supervision lifts CLAM's Dice from 0.215 to 0.350 with 10% annotated slides. That is a strong, cheap fix, and our method is orthogonal to it. We will run both methods with and without the 10% annotation regime to check whether the gains stack or overlap.
Proposed Experiments
The primary benchmark is Camelyon16, with its 270 training and 129 test slides and pixel-level tumor annotations. It supports slide AUC, localization Dice, FROC, and a breakdown by metastasis size; micrometastases are where context should matter most. TCGA-NSCLC and TCGA-RCC test whether the method helps or harms subtyping, where AUCs are near ceiling. PANDA tests grading, where tissue architecture over long distances is diagnostic. BCNB-HER2 tests a harder biomarker task where HookMIL's 66.42% leaves headroom. All results use five seeds with the same splits as the rerun baselines.
The decisive check is on Camelyon16. The full method must beat both the embedding-level feedback ablation and HookMIL on localization Dice and FROC, with non-overlapping seed intervals. The shuffled-context control must remove most of that gain. Without both, the claim that slide context changes what the encoder sees is not supported.
The secondary decisive check is the slide-level swinging test. Modulated tiles should flip predictions less often than frozen tiles as visible context grows. If modulation instead increases flipping, it would mean context-conditioned encoding inherits the spurious-context sensitivity Tomaszewska et al. documented. We would treat that as a negative result on stability, regardless of any accuracy gain.
References
- Attention-based Multiple Instance Learning with Mixed Supervision on the Camelyon16 Dataset
- Does context matter in digital pathology?
- Flexible Style Image Super-Resolution Using Conditional Objective
- Exploiting Spatial Dimensions of Latent in GAN for Real-Time Image Editing
- HookMIL: Revisiting Context Modeling in Multiple Instance Learning for Computational Pathology
- TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification
- MambaMIL+: Modeling Long-Term Contextual Patterns for Gigapixel Whole Slide Image
- Representation Learning of Histopathology Images using Graph Neural Networks
- Unlocking adaptive digital pathology through dynamic feature learning