A Cooperative Mixture Posterior over Adapters for Few-Shot Medical Segmentation Under Scanner Shift

Suppose a hospital has a large pretrained 3D segmentation network and wants to use it on a new scanner or protocol, but can annotate only one to ten scans from that site. The usual recipe freezes the big network and trains a small add-on module, an "adapter," on the few labeled scans. With so little data, many different adapters fit those scans equally well, and on new patients from the shifted site they can disagree sharply. Current methods keep one of them and discard the rest. We propose to keep several on purpose.

The method is to learn a small mixture of adapter solutions jointly, so that the components are pushed to cover different ways of explaining the support scans. Where these components disagree on a new scan, that disagreement becomes a map of where the adapted model is likely to be wrong. Because adapters are tiny, this kind of mixture is cheap here, whereas in deep generative models it has been expensive. The question the work answers is whether a cooperative mixture flags errors under scanner shift better than a set of independently trained adapters with the same compute. That ensemble is the obvious cheap competitor, and the whole proposal is designed around beating it or honestly failing to.

Core Claim

Our claim is that in few-shot adaptation of a frozen 3D segmentation backbone to a shifted imaging site, the posterior over adapter weights has several distinct modes. A variational posterior is an approximate probability distribution over unknown weights, fitted by optimization. When the posterior has several modes, a single-Gaussian variational posterior either collapses onto one mode or smears across all of them. This is the classic failure of unimodal and factorized approximations described by Bishop et al..

Independent ensembles of adapters are also likely to under-cover these modes, for a specific reason. Adapters are initialized near the identity and sit on a strong frozen backbone, so different random seeds tend to land in the same basin. A mixture trained jointly under an objective whose entropy term is the log of the mixture density rewards components for occupying different regions.

We predict three things at one to five support scans. First, error detection from component disagreement will be higher than for an equal-size independent ensemble. Second, there will be fewer "confident errors," meaning wrong voxels with low predicted uncertainty. Third, Dice overlap will be at parity with the other methods. We expect the advantage to shrink by ten support scans, as the posterior concentrates.

What Prior Work Supplies and Leaves Open

The method comes from the mixture variational inference literature. Kviman et al. show that jointly optimized mixture components in variational autoencoders "cooperate": they tile the target density instead of piling onto the same easy mode as independently trained ensembles do. Their approach cost 52 GPU-hours for four components because each component is a full encoder. Simkus and Gutmann show that stratified sampling, which draws from every component at every step rather than sampling a component at random, stabilizes the gradients of mixture bounds. We reuse both ideas, but apply them to a posterior over adapter weights rather than over latent codes. Because each component is only an adapter, the linear-in-components cost becomes minutes rather than days.

The adaptation setting comes from FSEFT. That work trains 3D convolutional spatial adapters on a frozen supervised Swin-UNETR and adds a transductive organ-size constraint, outperforming full fine-tuning at one to ten scans. Its adapters are deterministic, so it cannot report where it is unsure. Its size prior is also unimodal, which is a known risk for pathological anatomy.

Two neighbouring works use mixtures but address different problems. FedBEns builds mixtures of Laplace approximations over whole networks for federated learning. It finds that multi-modal posteriors beat unimodal ones: about 51.4% versus 37% on CIFAR-10 under strong heterogeneity. However, it needs curvature estimates for the full network and finds its modes by independent restarts, which is exactly the ensemble mechanism we test against. GMDI places a Gaussian mixture over domain descriptors but requires domain labels and abundant data. The shared-latent mixture translation model uses a latent mixture to preserve small structures when translating between CT contrast phases. That is image preprocessing, not uncertainty about a segmenter.

Two further papers motivate the problem and also constrain it. Avoiding Catastrophic Referral Failures finds that Bayesian networks make confident false positives under covariate shift. This is a direct warning that a posterior can be multi-modal and still miss errors. Ouyang et al. calibrate out-of-domain MRI segmentation post hoc using test-time augmentation and a shape autoencoder. They note that their shape prior assumes regular anatomy. We treat both as the bar that a principled posterior must clear.

Research Program

We keep the backbone frozen and place a posterior over the adapter weights $\phi$. The posterior is a mixture $q(\phi)=\sum_{k=1}^{K}\pi_k\, q_k(\phi)$. Here $K$ is between 2 and 6, each $q_k$ is a diagonal Gaussian over one full adapter, and $\pi_k$ are learned mixture weights. The prior $p(\phi)$ is a zero-mean Gaussian, which corresponds to the identity adapter. We train with the stratified mixture bound

$$ \mathcal{L}(q)=\sum_{k=1}^{K}\pi_k\,\mathbb{E}_{q_k(\phi)}\Big[\log p(\mathcal{D}_s\mid\phi)+\log p(\phi)-\log\sum_{j=1}^{K}\pi_j q_j(\phi)\Big], $$

where $\mathcal{D}_s$ is the set of one to ten labeled support scans and $\log p(\mathcal{D}_s\mid\phi)$ is the voxel-wise segmentation log-likelihood. The last term is what makes components cooperate. A component loses reward for sitting where the other components already sit. Independent training has no such term.

At test time, each component produces a segmentation. The prediction is their weighted average, and the uncertainty map is the mutual information between the component index and the voxel label. This is the part of the uncertainty that comes from components disagreeing, as opposed to noise that all components share. We also report total predictive entropy. All methods use FSEFT's transductive size constraint in the same form, so that it is not a confound. As a secondary analysis, we give each component its own size target, estimated from the support-scan loss it attains. This tests whether a mixture of anatomical size hypotheses helps on pathological cases where one target is wrong.

Two diagnostics come before any accuracy claim. The first asks whether the posterior is multi-modal at all. We measure functional diversity as pairwise Dice disagreement between components on target scans, and compare it with the disagreement between independently seeded adapters. The second asks whether disagreement falls on errors or on harmless boundary voxels.

Baselines

The naive reference point is a single deterministic FSEFT adapter trained on the same support scans, with no uncertainty beyond its softmax entropy. This is the method we are trying to make safer.

The strongest competitor is an independent ensemble of $K$ FSEFT adapters, trained from different seeds and different support-set orderings. It is the obvious cheap alternative, and since each adapter trains in 4–8 minutes on one GPU according to FSEFT, its cost matches ours almost exactly. To keep the comparison fair, it gets the same $K$, the same total optimizer steps, the same backbone, the same size constraint, and the same hyperparameter search budget. It also serves as the key ablation: keeping the Gaussian components but dropping the cross-component entropy term turns our method into exactly this ensemble, so any difference isolates the effect of cooperation. For absolute few-shot Dice, the file reports no per-organ numbers for FSEFT at one or ten scans: no reported result found on Lacuna for FSEFT absolute Dice at K=1–10. The only figures it gives are an improvement of up to +0.9% Dice from the size constraint and a +11.8% Dice gain from supervised over self-supervised pretraining. We will therefore reproduce FSEFT ourselves.

The unimodal Bayesian baselines are a single diagonal Gaussian variational posterior over the adapter, trained with the same bound at $K=1$, and a Laplace approximation fitted at the deterministic adapter's solution. These test whether multi-modality matters, or whether any posterior over the adapter would do.

The post-hoc baselines are voxel-wise temperature scaling and the calibration network of Ouyang et al.. On bSSFP-to-LGE cardiac MRI, that paper reports an expected calibration error (ECE) of 15.37%, against 16.78% for local temperature scaling. On the same MS-CMRSeg transfer, "beating it" means lower ECE than 15.37% at matched Dice. Beating the ensemble means a statistically reliable gain in voxel error-detection AUROC and a lower confident-error rate at one to five support scans, with Dice no worse than one point below it.

Why not just…?

Why not just train an ensemble of independent adapters? This is the real threat. We could find that seeds alone already produce diverse adapters, in which case our cooperation argument, borrowed from Kviman et al., would not transfer from latent codes to adapter weights. FedBEns's gains also came from independent restarts, which shows that restarts can find distinct modes in full networks. Our hypothesis is that near-identity adapters on a frozen backbone restart into the same basin. The functional-diversity diagnostic and the equal-compute ablation settle this directly. If independent and joint training tie on diversity and error detection, we will report that the cheap ensemble suffices and drop the claim.

Why not just use a unimodal posterior such as Gaussian variational inference or a Laplace approximation on the adapter? These are cheaper, and if the adapter posterior is effectively unimodal they will match us. The $K=1$ versus $K>1$ comparison under the identical bound tests exactly this. A clear win for $K>1$ only at the smallest support sizes would support the multi-modality claim. A tie would refute it.

Why not just calibrate post hoc, or rely on nnU-Net-style augmentation? Post-hoc temperature maps rescale confidence but cannot express that two anatomically different segmentations are both plausible. Catastrophic Referral Failures also needs a labeled out-of-domain validation set to fit its shifts, which a one-to-ten-scan site lacks. Heavy augmentation has not closed the gap on its own. AMOS reports MRI drops of over 20% mDice on unseen scanners, and FeTA 2024 still finds site effects of about 0.075 Dice despite universal augmentation. We will nonetheless include Ouyang's calibrator, and apply it on top of our mixture as well. If the combination is best, the fair conclusion is that the two approaches are complementary.

Proposed Experiments

The first setting follows the FSEFT recipe on AMOS. We pretrain a Swin-UNETR on the 500 CT scans, then adapt it to MRI with one, five, or ten labeled MRI scans drawn from a subset of scanners. We test on MRI from held-out scanners, which is where AMOS reports its largest collapse. The second setting uses FeTA, following FeTA 2022 and FeTA 2024. We pretrain on KISPI and Vienna and adapt with a few scans from an unseen center, including the 0.55T low-field KCL set. For context, top fully supervised methods reached Dice 0.86, HD95 1.69, and Euler characteristic difference 6.26 on that low-field set. The third setting uses MS-CMRSeg bSSFP-to-LGE for the calibration comparison.

We report Dice, 95th-percentile Hausdorff distance, and Euler characteristic difference, since overlap alone hides topological failures. For uncertainty, we report ECE, AUROC for detecting erroneous voxels from uncertainty, the confident-error rate, and scan-level risk–coverage curves for referring the least certain scans to a human. Every configuration is run over twenty random draws of the support set, because few-shot results depend heavily on which patients are chosen.

The decisive check is the equal-compute comparison against the independent ensemble at one and five scans on AMOS unseen-scanner MRI. The second check is whether confident errors persist. This would be the failure that Catastrophic Referral Failures predicts if the errors come from backbone features that no adapter posterior can see. If confident errors concentrate where all components agree, that locates the limit of adapter-level uncertainty and becomes a finding in its own right.