Routing by Loss Disagreement: Protecting Anatomy in Cross-Modality Image Synthesis

Medical image translation models turn one kind of scan into another, for example a non-contrast CT into a contrast-enhanced CT, or a T1-weighted MRI into a T2. They are usually trained with several losses added together. A pixel loss rewards matching intensities, and structural losses (edge, SSIM, or the Dice overlap of a frozen segmenter run on the output) reward keeping boundaries in the right place. These losses mostly agree. Where they disagree, the network has to compromise, and the compromise often shows up as a thin vessel or tumor margin that is either blurred away or hallucinated. Our hypothesis is that this disagreement is spatially concentrated near fine structure. If so, a single shared set of weights pays for it everywhere, and a fixed scalar loss weight cannot fix it.

We propose a mixture of experts, meaning a layer made of several parallel sub-networks plus a small gating model that decides which sub-networks process each spatial location. The gating model is trained to send locations where the pixel-loss and structure-loss gradients point in opposite directions to dedicated "routed" experts. An always-on "shared" expert then learns only from locations where the objectives agree. Our bet is that the shared expert ends up holding a modality-independent anatomical prior that transfers to unseen modality pairs, while the routed experts absorb the modality-specific tension between intensity fidelity and boundary fidelity. When we adapt to a new modality pair, we update only the routed experts.

Core Claim

The claim is that per-location conflict between loss components is a better routing signal than image content or edge strength. Training the shared expert only on agreement regions should improve boundary fidelity on held-out modality pairs, compared with dense models trained on the same multi-component loss. The new element is the routing criterion. Prior mixture-of-experts restoration models route by content, degradation type, or frequency. Prior multi-component losses fix their weights. No work in our file uses disagreement between loss terms to decide which parameters a location updates.

Relation to Prior Work

Three restoration systems already combine experts with detail preservation, and they supply most of our architecture.

  • AdaptIR builds heterogeneous experts (local convolution, Fourier, and channel gating) and blends them with a soft per-module weight under a plain pixel loss on 2D natural images.
  • M2Restore routes edge-dense regions to a convolutional branch and smooth regions to a state-space branch. It trains with L1 plus a load-balance loss on weather removal.
  • HiFi-MambaV2 uses a shared expert plus per-pixel top-1 routed experts over a Laplacian frequency split, for within-MRI k-space reconstruction.

We reuse the shared-plus-routed layout of HiFi-MambaV2, which echoes the basic/refinement split described for language models. We depart from all three in what the gating model is trained to predict. They also never test transfer across imaging physics.

The routing signal comes from STGC. In a vision-language mixture of experts, STGC flags tokens whose gradient opposes their expert's mean gradient, and penalizes the gating model for sending them there. It approximates gradients cheaply through layer biases. STGC measures conflict between tokens under a single loss. We measure conflict between loss components at a single location, which is a different axis, and we apply it to pixel synthesis.

The loss side is grounded in three synthesis papers:

  • 2S-CEDiff refines diffusion-generated CECT with fixed-weight L1, z-axis SSIM, and volumetric SSIM.
  • SINCO adds a frozen U-Net's Dice loss to pixel fitting and shows it preserves tumor margins. - Flow-based deformation guidance balances four empirically weighted terms for T1-to-T2 translation.

All three are dense networks with static weights. They motivate the proposal, because each needs a structural term that the pixel loss fights.

Two papers act as constraints. The classification-distortion-perception tradeoff shows that restoration cannot create task-relevant information absent from the input. We therefore treat segmenter Dice on synthesized images as a measure of preservation, never as an improvement over the source. Modality-partitioned expert models such as M4oE and MoME assign experts statically per modality, and they target segmentation rather than synthesis. Our experts are not tied to modalities, which is what lets them apply to an unseen pair.

Mechanism

Let $h_n$ be the feature at spatial location $n$ entering a mixture-of-experts layer. Let $L_{pix}$ be the L1 loss and $L_{str}$ the structural loss, which combines a gradient-magnitude edge loss with SINCO-style Dice from a frozen segmenter. We compute the per-location conflict

$$c_n = \cos\left(\frac{\partial L_{pix}}{\partial h_n}, \frac{\partial L_{str}}{\partial h_n}\right),$$

which costs one extra backward pass through the decoder to the feature map, not to the weights. The layer output is $y_n = S(h_n) + \sum_e r_{n,e} E_e(h_n)$. Here $S$ is the shared expert, $E_e$ are $k$-sparse routed experts, and $r_{n,e}$ are gating weights.

The gating model also outputs a scalar $s_n \in [0,1]$, which scales the contribution of the shared expert's gradient at location $n$. It is trained with an auxiliary loss of the form

$$L_{route} = \sum_n \big[\,\mathbb{1}(c_n < 0)\, s_n + \mathbb{1}(c_n \ge 0)(1 - s_n)\big] + \beta L_{bal},$$

where $L_{bal}$ is the standard load-balance term and $\beta$ is its weight. In words, the shared expert stops learning where the losses fight, and the routed experts take over there. Because $c_n$ is recomputed at every step, the gating model learns to anticipate conflict from content alone. That matters at test time, when no gradients are available.

On a new modality pair, $S$ is frozen, and only the routed experts and the gating model are fine-tuned on a small number of paired volumes.

Research Program

We first test whether the premise holds. In a dense U-Net trained on 2S-CEDiff-style losses, we map $c_n$ and measure two things. The first is how concentrated the negative-cosine mass is in a boundary band around segmenter contours. The second is whether that concentration correlates with the locations of vessel or margin errors. If conflict is diffuse, the method has nothing to exploit, and we stop.

Next, we build the conflict-routed model on a shared encoder-decoder backbone. The training mix contains paired translations from BraTS (T1 to T2) and HeartCT (non-contrast to contrast-enhanced CT). We hold out two settings:

  • Same physics, new anatomy and scanner: Coltea-Lung-CT-100W, evaluated zero-shot and after few-shot adaptation.
  • New physics: ReMIND intraoperative ultrasound to T2, adapted with about ten paired volumes.

The analysis has three parts. We probe whether the shared expert's features are more modality-invariant than a dense model's, using linear separability of modality identity. We check whether routed experts specialize by modality or by structure type. We track whether the shared expert's gradients really agree across losses during training.

Baselines

The naive reference is a dense U-Net of matched active parameters trained with L1 alone. It shows how much structure the pixel loss erases on its own.

The strongest existing dense competitor is a 2S-CEDiff-style model: the same backbone trained with L1, z-axis SSIM, volumetric SSIM, and our structural term, using fixed weights tuned by grid search. On HeartCT, 2S-CEDiff reports 29.87 dB PSNR and 0.891 SSIM (2S-CEDiff). No BraTS2021 T1-to-T2 figure for it was found on Lacuna. For BraTS2019 T1-to-T2, the flow-deformation model reports SSIM 0.819 against 0.783 for CycleGAN (Flow-based deformation guidance). For ReMIND iUS-to-T2, MMHVAE reports SSIM 98.5% and LPIPS 1.1% (MMHVAE), though that comes with full paired training rather than our few-shot regime.

The strongest mixture-of-experts competitor reproduces HiFi-MambaV2's shared-plus-routed layout, with content-based top-1 routing, a load-balance loss, and our full loss. It differs from our method only in the routing objective.

The key ablations hold the architecture fixed and change only the routing signal:

  • random routing at the same fraction of locations;
  • routing by edge magnitude, as in M2Restore's edge gate;
  • routing by high-frequency energy, as in a Laplacian split.

Every model gets the same data, the same number of active parameters and FLOPs, the same training steps, and the same 20-trial hyperparameter budget. Seeds are reported over three runs.

Beating the baselines means three things on the held-out settings, all at statistically equal global PSNR:

  • higher SSIM within a boundary band;
  • higher Dice of the frozen segmenter on synthesized images;
  • lower vessel-fragmentation counts.

A global PSNR gain alone does not count.

Why not just…?

Why not just learn the loss weights adaptively, for example by uncertainty weighting? Adaptive weighting changes one scalar per loss term. If conflict is concentrated at boundaries, a global scalar must either over-weight structure everywhere, which costs intensity fidelity in flat tissue, or under-weight it at margins. We include a learned-weighting dense baseline. If it matches our method in the boundary band, the spatial-routing story fails, and we report that.

Why not just apply per-location gradient projection to a dense network, removing conflicting components as gradient-surgery methods do? This is the most serious threat, because it uses the same signal we do. It resolves direction but adds no capacity: the projected gradient still updates the same weights everywhere. Our prediction is that it matches us within the training modalities but transfers worse, because no part of the network is kept free of conflict. We run it as a dense baseline at matched FLOPs. If it also matches us on Coltea-Lung and ReMIND, we will report the method as an expensive equivalent of projection.

Why not just route by edges or frequency, which is cheaper and probably correlated with conflict? If conflict were nothing more than edge strength, M2Restore's edge gate or HiFi-MambaV2's frequency split would already capture it. The edge-routing and frequency-routing ablations test exactly this. The phase-one map also measures how much conflict mass falls outside high-gradient regions, for example low-contrast lesion interiors. If edge routing ties with conflict routing, the gradient computation is not worth its cost, and we will say so.

Why not just fine-tune a pretrained dense model on the new modality? We compare full fine-tuning and adapter fine-tuning in the AdaptIR style, under the same few-shot budget, against routed-only adaptation. The claim that the shared expert holds a transferable prior stands or falls on this comparison.

Decisive Checks

The idea is worth pursuing only if four things hold:

  • Negative-cosine locations are spatially concentrated near structure.
  • Conflict routing beats edge routing and dense gradient projection on boundary-band SSIM and segmenter Dice on at least one held-out setting, with global PSNR within noise of the 2S-CEDiff-style baseline.
  • Freezing the shared expert during adaptation beats full fine-tuning on ReMIND with ten volumes.
  • The gating model, at test time and without gradients, predicts conflict locations with clearly better-than-edge-detector accuracy.

Failing the second check while passing the first would mean the signal is real but redundant with simpler gates. Failing the third would mean the specialization does not transfer.