Replaying Anatomy, Not Pixels: Breaking the Self-Training Loop in Multi-Site Medical Segmentation

When a hospital's segmentation model is updated scanner by scanner, it forgets the scanners it saw earlier. Privacy law usually forbids keeping old patient scans, so the standard fix is generative replay: a generator (a model that produces synthetic images) imitates past sites, and its samples are mixed into training at each new site. There is a catch. To keep one compact generator current, practitioners retrain it on its own earlier samples plus the new site's data, which puts the generator in a self-consuming loop. Such loops are known to erase rare cases and to drift away from real data. They are also expensive, since diffusion replay alone costs roughly ten times the compute of simpler continual-learning methods.

This note proposes a different split. In segmentation across scanners, the thing that shifts between sites is mostly appearance: intensity, contrast, noise, and resolution. The thing the model must not forget is mostly anatomy: which organs exist, their shapes, and their topology. We propose to keep anatomy in a fixed bank of label maps that is never regenerated. Only appearance is replayed, rendered cheaply from a few stored intensity statistics per site. Because nothing is ever refit on its own output, the self-consuming loop is cut. The replay becomes cheap enough to run on 3D volumes, and the study directly tests the claim that recursive replay silently erodes small organs and topology across hospital hops.

Core Claim

The evidence for the loop problem is strong but comes from other settings. On the Dangers of Bootstrapping Generation for Continual Learning and Beyond retrains StyleGANv2 and diffusion generators over up to ten generations on CIFAR and Flower102. By the tenth generation, a classifier trained on the synthetic images performs at chance. A linear probe separates real from synthetic features with over 99.9% accuracy, and this holds even for diffusion replay DDGR from the first increment onward. That paper also finds that label and class structure drift faster than image quality.

In medicine, AI-generated data contamination erodes pathological variability and diagnostic reliability shows recursive generation deleting rare findings. Pneumothorax disappears, report vocabulary shrinks by 98.9%, and false reassurance rises from 13.3% to 40.3%. That study concerns passive data contamination rather than replay pipelines. Hazard Compression shows diffusion replay buffers pruning rare boundary states in reinforcement learning.

These papers motivate the proposal. They do not test the setting where it would matter clinically: sequential adaptation across scanners on 3D segmentation, measured with small-organ and topology metrics.

Our claim has two parts. First, a single generator retrained recursively across $K$ site hops will lose small, thin, or topologically complex structures faster than it loses global image realism. Overlap scores such as Dice will largely hide this loss, and a topology score will reveal it. Second, freezing anatomy in a non-recursive label-map bank and replaying only per-site appearance will keep that loss flat in $K$. It will do so at a small fraction of the compute of learned generators and with storage that grows only by a handful of numbers per site.

The anchor is the finding from the Dangers paper that labels drift first. If the fragile component is structure, the structure is what should never pass through the loop.

Research Program

The system has three parts.

Anatomy bank. This is a fixed set of 3D label maps collected once at the first site. As a privacy-graded alternative, it can be replaced by a shape generator trained once on those masks and then frozen. Either way it is never retrained on synthetic output, so no generation-on-generation fitting occurs.

Per-site appearance record. When a site is seen, we store only per-organ intensity statistics: mean, covariance, and histogram quantiles. We also store a noise and blur profile and the slice spacing. This is in the spirit of storing class-conditional feature statistics, as TRIPS (Koniusz et al.) does for features and CCSI (CCSI) does by preserving normalization statistics across tasks. Here the statistics sit in image space and are tied to anatomy.

Renderer. At replay time, a label map is drawn from the bank and painted with the appearance record of a past site. This costs essentially nothing compared with diffusion sampling.

We will compare this design against learned-generator replay in both of its common forms. The first is recursive: one generator refit at each hop on its own samples plus new data, as in serial cyclical training with local VQ-VAE-2 models in Handling data heterogeneity with generative replay in collaborative learning for medical imaging. The second is non-recursive: one frozen generator per site.

The analysis tracks four quantities after every hop:

  1. Tail survival. This is the fraction of replayed volumes that still contain small organs such as the adrenal glands and duodenum, which AMOS singles out as fragile, together with their size distribution.
  2. Topology. This is the Euler Characteristic Difference (ED), a count-based score of spurious holes and disconnected pieces introduced by FeTA 2024. It is computed both on the replayed label structure and on the downstream model's predictions.
  3. Representational separability. This reuses the real-versus-replay linear probe from the Dangers paper.
  4. Cost. This is measured as GPU-hours and bytes stored per hop.

The honest limitation is that a fixed anatomy bank cannot represent anatomy that first appears at later sites, such as a new population or a new pathology. We measure this by holding out an anatomically distinct site. We also test a hybrid in which new label maps are appended to the bank when labels arrive, which is still non-recursive.

Baselines

The naive reference point is sequential fine-tuning with no replay. Joint training on all sites is the upper bound. In the chest X-ray continual study Fairness Evolution in Continual Learning for Medical Imaging, these reach AUC 0.55 and 0.78 respectively on CheXpert. That paper is classification, however, and for multi-site continual 3D segmentation we found no reported result on Lacuna. The closest reference numbers are static. AMOS reports 89.04% mDice for 3D-UNet on CT and a drop of more than 20% mDice on unseen MRI scanners AMOS. FeTA 2024 reports a winning ED of 20.9 and Dice 0.86 on its 0.55T low-field test set. Beating naive fine-tuning means recovering most of the gap to joint training on earlier sites without losing accuracy on the current site.

The strongest data-free competitor is recursive learned-generator replay, and it is run in two variants. The VQ-VAE-2 serial replay of the collaborative-learning paper was the method designed for exactly this cross-hospital setting. A DDGR-style diffusion replay is included wherever compute permits, because it is the highest-fidelity learned replay available. CCSI-style model inversion is a third data-free competitor. For fairness, all methods use the same 3D-UNet segmenter, the same site order, the same per-hop training epochs, and the same hyperparameter-search budget. Learned generators additionally receive their own tuning budget, reported separately, so that their cost is visible rather than hidden. The proposal beats these baselines if it matches or exceeds them on earlier-site mDice and has clearly lower ED and higher small-organ Dice at hop $K$, at a fraction of their GPU-hours.

LwF is the regularization reference because it has no generation cost at all. A small real-scan buffer serves as a privacy-violating upper reference. It is not a legitimate competitor, but it tells us how much the privacy constraint actually costs.

Three ablations isolate the key ingredient:

  • Refit ablation: we let the anatomy bank be refit on replayed masks at each hop. This deliberately reintroduces recursion and tests whether the loop, and not the rendering style, drives the effect.
  • Appearance ablation: we drop the per-site appearance records and render with random intensities.
  • Frozen-generator ablation: we replace the renderer with per-site frozen learned generators.

Why not just…?

Why not just keep one frozen generator per site, which already removes the loop?

This is the most serious alternative, because it removes recursion just as our design does. Its costs are elsewhere. Storage and training compute grow with one full 3D generator per site. At DDGR's scale of roughly 75,000 seconds on CIFAR-100, against roughly 8,000 seconds for regularization methods, per-site generator training on volumes is expensive. Learned generators may also underproduce rare structures even without recursion. We include it as an ablation. If it matches our tail and ED numbers, the contribution narrows to a large reduction in cost, and we will report it that way.

Why not just skip replay and render label maps with random intensities, so the model becomes robust to any scanner?

Random-intensity rendering from label maps is a real threat, because it makes the model robust to scanner appearance without remembering any particular site. The file offers no direct evidence on it in this setting. Against it stands AMOS, where heavy augmentation still left a drop of more than 20% mDice on unseen MRI scanners, and FeTA, where the acquisition site alone cost about 0.075 Dice despite augmentation. The appearance ablation settles the question: if random rendering matches site-specific replay, the per-site records are unnecessary. That would still be a useful and cheaper result.

Why not just use LwF or other regularization, which has no generation cost and no loop?

LwF has neither a generation cost nor a loop, so it is the right cost reference. It constrains the model's outputs without showing it the old appearance. In Fairness Evolution, LwF also opened a larger gender gap (0.028 versus 0.001 for pseudo-labeling), though that result is from classification. Our design should approach LwF's cost. If LwF matches us on earlier-site accuracy and ED, replay is not justified.

Why not just store a few real scans?

Privacy rules forbid it in the setting the proposal targets, as CCSI and InvCoSS both argue. We still run the buffer so that the cost of the privacy constraint is measured, not assumed. Storing label maps is itself a privacy question. For that reason the frozen shape-generator variant is part of the main comparison and not an afterthought.

Proposed Experiments and Decisive Checks

On AMOS, we order the eight scanners into a sequence of hops, first within CT, then crossing CT to MRI, and report per-organ Dice, ED, and cost after each hop. On FeTA 2024, three sites (KISPI, Vienna, and KCL at 0.55T) give a short sequence rich in topology, where ED is native. Retina and bone-age data from the collaborative-learning paper provide a classification sanity check against its own reported gains (4.88% accuracy and a roughly 50% MAE reduction).

The decisive check is a curve of small-organ Dice and ED against hop count $K$. Recursive replay should decline even while its mDice stays close to our method's. Our method and its refit ablation should diverge in exactly the way that isolates the loop. If recursive replay does not degrade over eight hops on AMOS, the motivating hazard does not transfer to scanner shift at this scale, and we will report that as a negative result. A second check is whether the real-versus-replay linear probe, which exceeds 99.9% accuracy for learned generators on natural images, stays lower for rendered replay. A third check is whether appearance records beat random rendering on held-out earlier scanners.