Cached-Bag Training: End-to-End Survival Learning from Whole Patients, Not Single Tiles

Medical prognosis models often have to predict when something bad will happen, such as death or cancer recurrence, from images far too large to fit on a GPU. Examples include gigapixel pathology slides and 3D CT volumes. The standard workaround has two stages. A frozen, pre-trained image network turns each small tile or slab into a feature vector offline. A small model then pools those vectors and is trained on a survival objective. Because the image network never sees the survival loss, it never learns what is prognostic. The two existing ways to let gradients reach the image network both give something up. One trains on isolated tiles, each labelled with its patient's outcome, and averages tile scores at test time. This loses the patient-level context that heterogeneous tumors need. The other freezes the backbone and tunes about 1% of its parameters.

We propose a third option, borrowed from video understanding. Keep a cached embedding for every tile of every patient. At each step, recompute only a few tiles with full gradients through the backbone. Then pool the fresh tiles together with the stale cached ones into a patient-level prediction, and train that prediction on a censored survival loss. The survival gradient reaches the backbone, the pooling model always sees the whole patient, and memory cost scales with the handful of fresh tiles rather than the size of the bag.

Core Claim

The bottleneck in end-to-end visual survival analysis is not only that the backbone is frozen. Every current fix also cuts the link between the per-tile gradient and the patient-level outcome. We claim that a cached patient bag, refreshed sparsely and with controlled staleness, recovers most of the benefit of full end-to-end training. It should beat frozen features, tile-level end-to-end training, and prompt tuning on discrimination at matched compute, without the calibration damage that ranking-driven end-to-end training is suspected to cause.

Where This Sits

Three papers supply the motivation. EE-Surv trains a full ResNet-50 with a Cox loss on individual 512-pixel tiles and averages tile risks per patient. Its risk score stays an independent predictor on colorectal cancer (multivariable p = 0.00265). It loses significance on heterogeneous gastric cancer (multivariate p = 0.0978). That is exactly the failure we would expect when tiles are trained without seeing each other. VPTSurv instead keeps the slide-level decoder and lets survival gradients reach only lightweight prompt adaptors in a frozen ViT. It improves C-index by 8.7–12.5% relative over two-stage baselines, but it never updates the backbone filters. AMIL-Surv is the clean decoupled reference: frozen ResNet-50 features, attention pooling (a learned weighted average over tiles), and a Cox loss. Our method keeps AMIL's patient-level pooling, keeps EE-Surv's full-backbone gradients, and drops the restriction that makes each of them a compromise.

The method itself is borrowed from video. ClipBERT shows that backpropagating through a few sparsely sampled clips beats dense frozen features: 22.0 vs 16.8 Recall@1 on MSRVTT retrieval. Its weakness is that each training step sees only a fragment of the video. E2E-LOAD shows how to extend context cheaply: cache earlier frame representations in a buffer and stop gradients through them. Combining the two ideas gives our cached bag, where sparse fresh tiles carry gradients and cached tiles supply context. The analogy has a limit we must test. Video caches are refreshed every few frames, whereas a tile in our bank may be revisited only once per epoch. Staleness is therefore a first-class variable here, not a detail.

Two papers constrain the design. TTE-Pretraining avoids the Cox loss's dependence on whole-cohort risk sets by using a piecewise-exponential discrete-time likelihood, which splits into independent per-patient terms. It reports large gains from survival-shaped representations. Yet it freezes the encoder at downstream adaptation, so whether end-to-end adaptation helps on top of such pretraining is still open. We adopt its discrete-time loss as the default rather than inventing a fix for small-batch Cox risk sets. Lillelund et al. show that most survival papers report only the C-index, a ranking metric blind to whether predicted probabilities are correct. DeepTCSR observed C-index gains alongside worse Integrated Brier Score. So calibration is a primary outcome here, not an afterthought.

Method

Each training patient $i$ has a bag of $N_i$ instances, meaning tiles of a slide or slabs of a CT volume, plus follow-up time and an event indicator. A memory bank stores one embedding $\tilde z_{ij}$ per instance, initialized by a full pass of the pre-trained backbone.

At each step we sample a mini-batch of patients and, for each, $k$ instances (for example $k = 16$ out of thousands). We pass those $k$ instances through the backbone $g_\theta$ with gradients to get fresh embeddings $z_{ij} = g_\theta(x_{ij})$. The patient representation is attention pooling over the full bag, mixing fresh and cached embeddings:

$$h_i = \sum_{j \in F_i} a_{ij}\, z_{ij} + \sum_{j \notin F_i} a_{ij}\, \mathrm{sg}(\tilde z_{ij}),$$

Here $F_i$ is the set of fresh instances, $a_{ij}$ are softmax attention weights computed over all $N_i$ embeddings, and $\mathrm{sg}$ denotes stop-gradient. A discrete-time hazard head on $h_i$ is trained with the censored negative log-likelihood, with right-censored patients contributing only survival terms up to their last follow-up. The fresh embeddings are then written back into the bank.

Gradients reach the backbone only through fresh instances. Each fresh instance's gradient is weighted by its attention share in the context of the whole patient, which is precisely what tile-level training lacks.

Two design choices need study.

The first is how to choose the $k$ fresh instances. Uniform sampling is unbiased. Sampling in proportion to the current attention weights concentrates backbone updates on the tiles that drive the prediction. We use a mixture of the two and treat the mixing weight as an ablation.

The second is staleness. We compare three bank-update rules. Plain write-back alone lets cached embeddings lag the backbone by up to an epoch. Write-back plus a background refresh from an exponential-moving-average copy of the backbone keeps the bank consistent, as momentum encoders do in contrastive learning. Periodic full re-extraction is the expensive reference. We measure staleness directly as the cosine distance between cached embeddings and those the current backbone would produce. The goal is a curve of performance against staleness, which tells practitioners how often they must refresh.

Research Program

The work proceeds in three stages.

The first stage builds the cached-bag trainer for pathology slides and for CT volumes, with slabs treated as instances. Both use the same backbone families as the comparison methods: ResNet-50 for slides, matching EE-Surv and AMIL, and the 3D encoders used by TTE-Pretraining for CT.

The second stage runs a matched-compute comparison against the alternatives below. Every method gets the same GPU-hours, the same hyperparameter search budget, and the same patient-level cross-validation splits.

The third stage is the calibration study. We ask whether end-to-end adaptation, under a Cox loss versus a discrete-time likelihood, changes Integrated Brier Score and D-calibration relative to frozen features. D-calibration tests whether predicted survival probabilities match observed event frequencies across the cohort.

The datasets are TCGA-CRC (413 patients) and TCGA-STAD (362 patients), the EE-Surv cohorts; STAD is the heterogeneous case where tile averaging failed. We add HECKTOR head-and-neck PET/CT, which has about 75% censoring, and INSPECT chest CT with its longitudinal outcomes.

Baselines

The naive reference is a clinical Cox proportional-hazards model on stage, grade and age. Any imaging model must add value over it. On HECKTOR, TMSS reports C-index 0.682 for classical CoxPH. On the lung tissue-microarray cohort, AMIL-Surv reports 0.61 ± 0.07 for the clinical model.

The decoupled baseline is AMIL: frozen backbone features with attention pooling, trained with the same discrete-time loss we use, so that loss choice is not confounded with end-to-end training. This is the baseline a careful skeptic expects to be hard to beat.

The tile-level end-to-end baseline reimplements EE-Surv: full backbone, survival loss on single tiles, mean aggregation of tile risks. EE-Surv reports log-rank p = 0.0021 and multivariable p = 0.00265 on TCGA-CRC, and multivariate p = 0.0978 on TCGA-STAD. No reported C-index on TCGA-CRC or TCGA-STAD was found on Lacuna, so we will compute it ourselves for all methods.

The parameter-efficient baseline reimplements VPTSurv's adaptor tuning on our cohorts. Its reported 59.93% and 61.10% C-index on IHC-CD4 and IHC-CD8 come from datasets we cannot access, so the comparison will be on shared data.

On HECKTOR the strongest reported end-to-end number is TMSS at C-index 0.763 (0.74 on hold-out), and we will compare against it there. On INSPECT, TTE-Pretraining reports only relative gains, not absolute values, so we will rerun its frozen-encoder protocol as the baseline and add our cached-bag fine-tuning on top of its pretrained encoder.

The key ablation removes the bank. Fresh instances are pooled alone, as in ClipBERT-style sparse training. This isolates the contribution of cached context.

To count as a win, the method must beat AMIL and EE-Surv on time-dependent C-index by more than the cross-validation standard deviation, at matched compute, and must not worsen Integrated Brier Score relative to AMIL.

Why not just…?

Why not just use frozen foundation-model features with a good MIL head? This is the real threat. If modern pretrained features are already prognostic, end-to-end adaptation may add noise on cohorts of a few hundred patients. The evidence that adaptation helps comes from VPTSurv's gains over frozen two-stage methods, and from video, where ActionFormer argues frozen features limit fine-grained boundary learning. Neither shows this for full backbones on small survival cohorts. Our matched-compute comparison against AMIL settles it. We will report a null result if that is what we find, and test whether any gain grows with cohort size by subsampling INSPECT.

Why not just do tile-level end-to-end training as EE-Surv did? Its own result on gastric cancer suggests averaging tile risks fails under heterogeneity, because each tile inherits the patient label regardless of whether it contains prognostic tissue. The bank-off ablation and the EE-Surv baseline together separate the value of end-to-end gradients from the value of patient-level context.

Why not just tune adaptors as VPTSurv does? Adaptor tuning is cheap and may be enough. We will run VPTSurv-style tuning inside the cached-bag trainer as well. That separates "which parameters get updated" from "whether the pooling sees the whole bag," so a win attributable only to full-backbone updates would be visible as such.

Why not just accumulate gradients over the whole bag with activation checkpointing? For small bags such as HECKTOR slabs this is feasible, and we will run it there as the exact reference that the cache approximates. For slides with thousands of tiles, it multiplies step time by the bag size, which our compute budget makes explicit.

Decisive Checks

Three outcomes would decide the question.

First, on TCGA-STAD the cached bag should beat both tile-level EE-Surv and frozen AMIL on C-index. If it does not, the context argument fails.

Second, performance should degrade smoothly with measured staleness, and EMA refresh should close most of the gap to full re-extraction. If plain write-back already matches full re-extraction, the refresh machinery is unnecessary and we will say so.

Third, end-to-end training with a Cox loss should show worse Integrated Brier Score than frozen features while the discrete-time likelihood does not. That would confirm the calibration concern raised by Lillelund et al. and DeepTCSR, and justify our default loss.