Diverse Experts That Tell Adversarial Adaptation Where to Stop

When a segmentation network trained on one hospital's scans is moved to another hospital, or from MRI to CT, its accuracy often collapses. The standard repair is adversarial domain adaptation: a second network, the discriminator, tries to tell source features from target features, and the segmenter is trained to fool it. This alignment pressure is applied uniformly across every pixel. At thin, low-contrast boundaries such as the left-ventricle myocardium, the subtle texture cues the segmenter needs are exactly the cues that differ across scanners, so the alignment pressure erases them.

We propose to put several small prediction heads, the "experts," on a shared encoder. Training makes the experts agree on the most likely class at each pixel but disagree, on purpose, about how they rank the remaining classes. On the unlabeled target images, the places where these deliberately diversified experts start to disagree about the top class form a map of where the domain gap is unresolved. We use that map to relax adversarial pressure at those pixels and to withhold pseudo-labels there. The claim is specific and testable. Without the diversity term, experts collapse into copies and their disagreement is uninformative. With it, disagreement becomes a cheap, single-network estimate of target error that improves both boundary accuracy and calibration.

Core Claim and Positioning

Three findings in the literature motivate the claim. First, adversarial alignment trades class separability for domain invariance, as The Invariance Discriminability Tradeoff in Domain Adaptation argues, and medical adapters show the predicted damage at low-contrast structures. Dou et al. adapt early layers of a frozen MRI segmenter to CT against a Wasserstein critic and report large Dice recovery but persistent boundary error on the myocardium. SIFA combines image translation with output-space adversarial alignment in one shared encoder. Both apply alignment pressure everywhere with a single model, and neither has an internal signal for where that pressure is hurting.

Second, mixtures of experts collapse into redundant copies unless something forces them apart. Advancing Expert Specialization for Better MoE shows that balancing how much traffic each expert receives does not produce functional difference. SimSMoE finds the collapse in representations even when outputs differ.

Third, the method comes from two papers. Learning a Mixture of Granularity-Specific Experts shows that full output divergence harms classification, so it penalizes expert similarity only over the non-top classes. That raises CUB-200-2011 accuracy from 85.4% to 89.4%. We transfer this class-masked divergence from images to pixels. Adversarial Experts Model for Black-box Domain Adaptation uses prediction discrepancy between two external classifiers, a black-box source model and CLIP, to locate hard target samples. Its experts are fixed outside models rather than trained heads, and it never feeds discrepancy back into how hard alignment pushes per pixel.

The closest MoE-for-shift work differs on both axes. GMoE relies on emergent specialization under plain training. HMOE relies on gating entropy and reports collapse at large expert counts. P-MoLE ensembles frozen site models after training. None of these uses adversarial alignment, and none uses diversity to steer it.

A paper that constrains the design is Toward Calibrated Mixture-of-Experts Under Distribution Shift. It finds that soft-routed mixtures become miscalibrated under shift because different expert combinations yield identical scores. We therefore default to uniform averaging of the heads and treat a learned per-pixel router only as an ablation. The contribution does not depend on routing.

Method

A shared encoder feeds $K$ lightweight decoder heads; we use $K=4$. Let $p_k(x)$ be expert $k$'s class distribution at pixel $x$, and let $\bar p(x)$ be their mean. Let $c^*(x)$ be the ground-truth class on source pixels, or the argmax of $\bar p(x)$ on target pixels. Let $q_k(x)$ be $p_k(x)$ with class $c^*$ removed and the rest renormalized. Each head is trained with ordinary cross-entropy on source labels. The diversity term pushes the experts' tail distributions apart while leaving the top class untouched:

$$\mathcal{L}_{div} = -\,\mathbb{E}_x\,\mathrm{JS}\big(q_1(x),\dots,q_K(x)\big),$$

where JS is the generalized Jensen–Shannon divergence. We use it because it is bounded, unlike KL, so maximizing it cannot diverge.

On target pixels we compute an agreement weight $a(x)\in[0,1]$. It is one minus the normalized disagreement among the experts' top-class predictions; we use the variance of $p_k(x)$ at $c^*(x)$ across heads. A pixel-wise discriminator $D$ on encoder features $F(x)$ is trained through a gradient reversal layer, which flips the sign of the gradient so the encoder learns to fool $D$. Its loss is weighted by agreement:

$$\mathcal{L}_{adv} = \mathbb{E}_{x\sim T}\,a(x)\,\log\big(1-D(F(x))\big) + \mathbb{E}_{x\sim S}\log D(F(x)).$$

Self-training pseudo-labels on the target are kept only where $a(x)$ exceeds a threshold.

The hypothesis behind the direction of the weighting is that confident, agreed regions are safe to align, while contested boundary regions are where invariance pressure destroys detail. The opposite policy, used by AEM, is to emphasize hard samples. That policy is a plausible competitor, and we test it directly by flipping $a$ to $1-a$.

Research Program

The work has three parts. The first is a diagnostic study, run before building the full method. It asks whether class-masked diversity makes target-domain expert disagreement predict target errors better than undiversified heads do. The measure is pixel-level area under the ROC curve for detecting segmentation errors. The study is run on a source-only model and on a plain adversarial model. If diversity does not improve this error-detection score, the method has no basis and we stop.

The second part integrates the agreement-weighted alignment and pseudo-label masking on two shifts. One is MM-WHS 2017 MRI→CT cardiac segmentation. The other is AMOS, using the CT→MRI direction and a split that holds out MRI scanners. AMOS reports that held-out MRI scanners cause a drop of over 20% mDice, even though 3D-UNet reaches 89.04% mDice on in-distribution CT.

The third part analyzes what the experts learn. We measure inter-expert representation similarity with centered kernel alignment, following SimSMoE. We map where disagreement concentrates, and we test whether it tracks the structures with known boundary failure: myocardium, pancreas, and adrenal glands.

Baselines

The naive reference point is the source-trained network applied to target data unchanged. On MM-WHS, Dou et al. report 14.3% average Dice for this setting, with 0.8% on the left-ventricle blood cavity. SIFA reports 17.2% for its own unadapted baseline. We rerun it with our encoder so the floor is measured in-house.

Dou et al. is the monolithic feature-level adversarial baseline. They report a +43.4% average Dice gain and 74.8% Dice on the ascending aorta. It shows what uniform alignment achieves without experts.

The strongest existing method is SIFA, which reports 73.0% average Dice on MM-WHS MRI→CT. Because SIFA also translates images, we compare in two ways. First, our method with feature alignment only is compared against SIFA's feature-alignment-only configuration. Second, our heads and agreement weighting are added on top of full SIFA, and the result is compared against SIFA alone. Beating SIFA means exceeding 73.0% average Dice when our module is added to SIFA, with a clear myocardium Dice and average surface distance improvement at matched training iterations. The feature-only variant is expected to close much of the gap to SIFA without the cycle-consistent generators.

For AMOS we found no reported adaptation result on Lacuna for the held-out-scanner MRI split. There we compare against our own reruns of 3D-UNet source-only, feature-level adversarial adaptation, and SIFA. All methods share the same backbone, augmentation, training iterations, and a hyperparameter search of equal size, 20 trials each, tuned on a small labeled target validation split that is reported separately.

The key ablation isolates the diversity term. We train identical $K$ heads with agreement weighting but without $\mathcal{L}_{div}$, so that disagreement comes only from random initialization. A second ablation uses unmasked full-distribution divergence, to show the masking matters. A third flips the weighting to $1-a$, the AEM-style emphasis on hard pixels. The final ablation is a learned soft router over the heads, which tests the calibration warning from Wong et al.

Why not just…?

Why not just use a deep ensemble of independently adapted networks, whose disagreement is diverse for free? This is the most serious threat. Independently seeded adversarial models are a well-known source of uncertainty. We run a $K$-member ensemble of feature-level adversarial models, using its disagreement for the same weighting, at both $K\times$ compute and matched compute; the matched version uses smaller members. If the full-compute ensemble matches our error-detection score and Dice, the honest conclusion is that our contribution is a cheaper single-network ensemble, and we will report it that way. Our expectation is that independent seeds disagree mostly in confident interiors, while masked diversity concentrates disagreement in the class tail at boundaries. The diagnostic study measures exactly that difference.

Why not just weight alignment by the entropy of a single model's prediction? Entropy is the zero-cost version of our agreement signal. Its weakness is that an overconfident model under shift produces low entropy exactly where it is wrong. The 0.8% Dice of the unadapted segmenter on one cardiac structure suggests confident failure is common. We include single-model entropy weighting as a baseline and compare error-detection scores and pixel-level calibration error. If entropy matches expert disagreement, the experts are unnecessary.

Why not just split the encoder into a shared aligned branch and a private unaligned branch, separating invariant and site-specific features without any experts? This directly addresses the tradeoff described by Chen et al. and avoids extra heads. It decides in advance which channels are protected from alignment, but it cannot decide which pixels are, and the damage we target is spatially localized at boundaries. We implement a shared-plus-private variant with the same parameter count. If it matches our boundary metrics, spatial targeting is not what matters, and the proposal's mechanism is refuted.

Proposed Experiments and Decisive Checks

The data are MM-WHS 2017 (MRI→CT and CT→MRI, standard 16/4 subject split) and AMOS (CT→MRI with held-out MRI scanners). If time permits, a secondary check uses PolypGen, the six-center polyp benchmark used by TransNetR. The metrics are per-structure Dice, average symmetric surface distance, pixel-level expected calibration error, and error-detection area under the ROC curve from the disagreement map.

Three checks are decisive. First, diversified heads must give higher target error-detection scores than undiversified heads and than single-model entropy; this is the precondition. Second, agreement-weighted alignment must improve myocardium, pancreas, and adrenal Dice and surface distance over uniform alignment, beyond seed variance across three seeds. Third, the benefit must survive comparison with the compute-matched ensemble and the shared-plus-private split.

A negative outcome on the first check would show that masked tail diversity does not produce useful disagreement in dense prediction. A negative outcome on the third would reduce the method to cheap ensembling. Either outcome is informative and would be reported.