A hospital's 3D segmentation model loses accuracy as scanners are replaced, protocols change, and annotation guidelines are revised. The natural remedy is meta-learning ("learning to learn"). The model is trained on many small tasks so that it can later adapt from a handful of labeled scans. Each past acquisition era is treated as a task, and the model learns to adapt to a new era from three or four annotated volumes. This proposal argues that the naive version of this plan fails silently, for a specific reason. Every past era has the same anatomy and the same label meanings, so one fixed network can solve all of them at once. The meta-learner therefore learns to ignore the few examples it is given. It collapses into an ordinary pooled model, and it has no adaptation ability left when a truly new scanner arrives.
The fix is to make the examples necessary. At training time, each episode receives a randomly drawn hidden "key", shared by that episode's example scans and its test scans. The key is a synthetic acquisition change (an intensity remapping, a resampling of the spacing between slices, a field-of-view shift) and, sometimes, a change in label definition (for example, whether a boundary structure counts as foreground). Because the key cannot be read from any single scan, the only way to segment an episode's test scans is to infer the key from its labeled examples. We then ask whether a 3D model trained this way adapts to real, later acquisition eras better than the simple alternatives: pooling and fine-tuning, sliding-window retraining, and training with the same augmentations but without episodes.
Core Claim
Meta-learning over temporal eras of volumetric scans is an extreme case of what the meta-learning literature calls non-mutually-exclusive tasks. In that setting the outer loop finds a single function that fits every training task, and the inner adaptation step becomes vestigial. Meta-Learning without Memorization documented this collapse on non-mutually-exclusive Omniglot, where vanilla MAML fell to 7.8% accuracy. Meta-Learning Requires Meta-Augmentation showed that augmentations which make labels unpredictable without the support set repair it. Those authors call these "conditional-entropy-increasing" augmentations; label shuffling moved MAML from 30.2% to 46.0% on non-mutually-exclusive Mini-ImageNet.
Label shuffling is meaningless for dense 3D anatomy, and mixing volumes produces impossible geometry. So the direction has had no working anti-memorization tool for CT and MRI. Our claim is that episode-consistent acquisition and definition keys play the same role for volumes. They raise the uncertainty of the label map given a single scan without breaking anatomical structure, because the same local intensity pattern maps to different labels under different keys.
We make two predictions. With keys, the benefit the model gains from its support scans stays large throughout meta-training. Without keys, that benefit decays toward zero. The keyed model should also adapt to real held-out scanner eras from few labeled volumes better than an equally augmented model trained without episodes. That second comparison is the one that matters, because it separates the meta-learning structure from the augmentation itself.
What Prior Work Does and Does Not Cover
The closest work is MetaMedSeg. It applies first-order Reptile to few-shot organ segmentation on the Medical Decathlon and samples related slices along the depth axis of one volume. We differ from it in three ways. It uses a 2D U-Net, which discards through-plane context. It never tests for memorization. And it draws support and query slices from the same volume, which is close to the "train-train" regime that A Representation Learning Perspective on the Importance of Train-Validation Splitting shows leads to high-rank memorization (Omniglot 89.0% versus 97.3% with a proper split). We use a native 3D backbone, support and query sets drawn from disjoint volumes, and an explicit memorization diagnostic.
Meta-Learning under Task Shift reweights training tasks by estimated density ratios toward the test distribution. It was evaluated only on static clustered 2D benchmarks. Its weights blow up exactly when a new scanner falls outside historical support, which is our regime, so we treat it as a constraint on design rather than a method to reuse.
On the temporal side, two papers motivate us and limit our ambitions. Evaluation of domain generalization and adaptation on temporal dataset shift found that invariance methods never beat pooled training on a decade of MIMIC-IV data. For that reason we do not try to make the model time-invariant; we try to make it quick to adapt. EMDOT showed that random splits overstate deployed performance and that sliding windows win in volatile periods. We adopt its backtesting protocol and its sliding-window baseline. Both studies covered tabular data and 2D radiographs only.
On the volumetric side, the main imaging motivation is the sacroiliitis MRI study, which reports degradation on 5- and 10-year follow-up scans after scanner upgrades but proposes no remedy. AMOS reports drops of over 20% on unseen MRI scanners and finds that a plain 3D U-Net (89.04% mDice on CT) beats Swin-UNetr (86.32%). That result justifies our choice of a convolutional backbone. Shortcut Learning in Medical Image Segmentation shows in 2D that center cropping and zero padding teach coordinate shortcuts. Our spatial keys (per-episode offsets of the field of view) are designed to deny the 3D analogue of those shortcuts to the outer loop.
Research Program
The backbone is a 3D U-Net. Small 3D convolutional adapter blocks are inserted after each decoder stage; the adapter design follows MA-SAM, but here the adapters are the only parameters updated in the inner loop. The outer loop learns the backbone and the adapters' initialization using first-order updates on 3D patches, which keeps memory tractable.
Each training episode is built as follows. We draw one historical era, meaning one scanner or site. We draw a key. We sample K support volumes and a disjoint set of query volumes from that era and apply the same key to all of them.
The key has three parts. The first is an acquisition part: a random, possibly non-monotonic intensity transfer curve, a resampling of slice spacing, noise level, and a field-of-view translation along all three axes. The second is an optional definition part, which applies morphological inclusion or exclusion rules to label boundaries, mimicking a guideline revision. The third is a severity parameter that we sweep.
The central diagnostic is the adaptation gain, tracked over training on held-out training-era episodes:
$$G = \mathcal{L}_Q(\theta) - \mathcal{L}_Q(\theta'_S)$$
Here $\mathcal{L}_Q$ is the segmentation loss on an episode's query volumes, $\theta$ is the meta-learned initialization, and $\theta'_S$ is the result of adapting the adapters on that episode's support set $S$. If $G$ tends to zero, the model has memorized a global solution.
The analysis has three parts. First, we measure $G$ with and without keys to test whether temporal meta-learning collapses the way the meta-augmentation literature predicts. Second, we ablate each key component (intensity, spacing, spatial offset, definition) to find which ones keep the inner loop alive. Third, we backtest adaptation to real later eras.
Baselines
The naive reference point is a 3D U-Net trained on all historical eras and deployed frozen. Every method must at least beat it on later eras, and it measures how much drift there is to recover.
The strongest simple competitor is pooled training plus fine-tuning. The same 3D U-Net is trained on all historical eras and then fine-tuned on the same K labeled volumes from the new era. Evaluation of domain generalization and adaptation on temporal dataset shift found that pooled training beat every invariance method, which makes it the incumbent to beat. We give it the same adapter parameterization, so that only the training regime differs, and also a full fine-tuning variant. Its tuning budget in trials and GPU-hours matches ours.
The sliding-window baseline follows EMDOT: the model is retrained on the most recent eras, and the window length is tuned by backtesting.
The key ablation is augmented pooled training plus fine-tuning. It applies exactly the same key distribution, but independently per scan, with no episodes and no inner loop, and then fine-tunes on K new-era volumes. This isolates whether the gain comes from the augmentations themselves or from forcing support-dependent adaptation. The second ablation is plain temporal meta-learning without keys, which should reproduce the memorization collapse. A third ablation replaces keys with inner-loop gradient noise from Regularizing Meta-Learning via Gradient Dropout. That method lifted cross-domain MAML from 45.56% to 51.05% on Mini-ImageNet to CUB, and it is the cheapest existing anti-memorization tool that transfers to dense prediction without modification.
For sequential multi-site prostate MRI we run the U-Net Expert Gate (UNEG) from What is Wrong with Continual Learning in Medical Image Segmentation?. It trains a separate frozen nnU-Net per domain and routes each scan to one of them with an autoencoder. It reports about 87% mean Dice on prostate MRI without domain labels, against about 70% for EWC and about 74% for MiB, which makes it the strongest reported number for the sequential setting. UNEG trains each expert on a full domain, not K volumes, so we report it at its native data budget and also restricted to K volumes per new domain. We also report memory growth.
For unseen-scanner transfer on AMOS, the reference is MA-SAM's reported 60.4% Dice on unseen MRI, where nnU-Net dropped to 10.9%. We rerun MA-SAM's adapters under our K-volume budget rather than relying on its zero-shot figure.
Beating a baseline means higher mean Dice on later eras at K of 1, 3 and 5, by more than the spread across three seeds and three era orderings, without lower Dice on earlier eras.
Why not just…?
Why not just pool all history and fine-tune on the new scans? This is the most likely winner, and the proposal is built around losing honestly to it if that is what happens. The case against it is indirect but specific. With K of 1 to 3 volumes, fine-tuning a pooled model starts from parameters that were never shaped to change in response to a few examples. The EMDOT staleness result suggests that high-capacity models drift fastest. The pooled-plus-fine-tune baseline, run with identical adapters and budget, settles the question directly. If it matches the keyed meta-learner at every K, the claim is false.
Why not just train with the same augmentations and skip meta-learning? This is the real threat. Domain randomization alone may teach robustness that looks like adaptation. The keyed-versus-per-scan-augmentation ablation exists to answer it. Our prediction is that per-scan augmentation teaches the model to average over keys, which blurs boundaries under strong non-monotonic intensity keys. Episode-consistent keys instead teach the model to resolve the key from the support set, which should show up as larger $G$ and a steeper improvement with K. If the two curves coincide, we will report that augmentation, not meta-learning, is the active ingredient.
Why not just retrain on a sliding window of recent data? EMDOT shows that sliding windows win when data are plentiful and the environment is volatile. Annotated 3D volumes from a new scanner are neither plentiful nor quick to obtain. The backtest shows where the crossover lies as the window's labeled volume count grows. We expect sliding windows to catch up once the new era has tens of labeled volumes, and we will report that point rather than hide it.
Why not just keep one frozen expert per scanner, as UNEG does? UNEG's roughly 87% Dice relies on a full labeled domain per expert and grows linearly in memory. It also transfers nothing forward from one expert to the next. Our K-volume comparison tests whether a single shared adaptable model reaches comparable Dice with a small fraction of the labeled data and parameters.
Proposed Experiments
There are three benchmarks. The first is the sequential six-source prostate MRI stream, with hippocampus and right-ventricle MRI, used in What is Wrong with Continual Learning in Medical Image Segmentation?. Sources are presented in order, and each new source is adapted with K labeled volumes.
The second is AMOS, with its 8 scanners and 2 centers. Scanners are ordered into a pseudo-temporal stream, and MRI scanners are held out to reproduce the over-20% drop reported there.
The third is the DESIR longitudinal MRI cohort from the sacroiliitis study. It is the only resource in our file with genuine 5- and 10-year scanner drift within the same patients. We use it if access permits, training on baseline scans and backtesting on follow-ups, with bone segmentation as the dense task.
The Medical Decathlon provides a direct comparison to MetaMedSeg's protocol; that paper reports 68.33% IoU on cardiac segmentation with 15 images against 58.78% for supervised training.
Metrics are Dice per era; Dice as a function of K; the adaptation gain $G$ over training; Dice on earlier eras after adaptation; and Dice as a function of the distance of target structures from the volume center, which probes 3D coordinate shortcuts.
Three results decide the question. The first is whether $G$ decays toward zero in keyless temporal meta-learning but not with keys. If $G$ does not collapse without keys, the memorization premise is wrong for this setting. The second is whether the keyed model beats augmented pooled training plus fine-tuning at small K on real held-out eras. The third is whether that advantage holds on DESIR's true longitudinal drift and not only on scanner-ordered pseudo-time.
References
- Meta-Learning without Memorization
- Meta-Learning Requires Meta-Augmentation
- MetaMedSeg: Volumetric Meta-learning for Few-Shot Organ Segmentation
- A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learning
- Meta-Learning under Task Shift
- Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine
- Evaluating Model Performance in Medical Datasets Over Time
- A deep learning model for the diagnosis of sacroiliitis according to Assessment of SpondyloArthritis International Society classification criteria with magnetic resonance imaging
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Shortcut Learning in Medical Image Segmentation
- MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation
- Regularizing Meta-learning via Gradient Dropout
- What is Wrong with Continual Learning in Medical Image Segmentation?