Keeping Loss Weights Out of the Router

Mixture-of-experts models (networks in which a small "router" decides which specialist sub-networks process each input) are increasingly used to train one model on many tasks at once. A common example in medical imaging is a pool of small-kernel experts for sharp boundaries alongside a wide-context expert for anatomy. These models still need a weighted sum of losses: one segmentation loss per task, plus a term that keeps the router from sending everything to one expert, plus boundary or auxiliary terms. Our hypothesis is that in routed models the loss weights do more damage than in ordinary shared networks. A weight does not only scale a gradient. It also tells the router which experts to favor. If one task's loss is large, the router drifts toward that task's preferred experts, those experts get more updates, and they become still more attractive. The result is a feedback loop that turns a modest hyperparameter mistake into expert collapse.

We propose to break that loop. The router learns from scale-normalized task losses that no balancing weight touches. The expert weights receive gradients rebalanced per expert and per task, using the router's own assignments to attribute each expert's gradient to the tasks that produced it. Because every task reaches every expert through a known gate, this rebalancing can be done with a gradient hook on each gate edge, with no extra backward passes. The practical payoff we test is twofold. Accuracy should stay flat across a wide range of loss weights. And that stability should make it possible to train a single multi-dataset segmentation model in one stage, rather than pre-training on source datasets and fine-tuning on the target.

Core Claim

In a task-routed network, sensitivity to loss weights comes mostly from the weights leaking into the router, not from conflict inside individual experts. We will show three things. First, removing the leak (router trained on normalized losses, experts trained on attributed and rebalanced gradients) reduces the spread of results across a loss-weight sweep by a large factor. Second, it does so at matched or better accuracy than applying an existing global balancer such as CAGrad or LDC-MTL to the same routed model. Third, the stabilized model trained in a single stage matches a two-stage pre-train-then-fine-tune pipeline.

Concretely, let task $t$ reach expert $e$ through gate value $g_{t,e}(x)$ for input $x$, and let $\theta_e$ be the weights of expert $e$. The expert update we use is

$$\Delta\theta_e \propto -\sum_t w_{t,e}\, \mathbb{E}_x\big[g_{t,e}(x)\,\nabla_{\theta_e}\mathcal{L}_t\big],$$

where $w_{t,e}$ is a per-expert, per-task weight. The rule for $w_{t,e}$ adapts the discrepancy-control idea of LDC-MTL from tasks to expert–task pairs: it equalizes the router-attributed losses $\bar g_{t,e}\,\mathcal{L}_t$ among the tasks that actually share expert $e$, where $\bar g_{t,e}$ is the average gate mass. A private expert, used by one task, gets weight 1 and is left alone. The router parameters receive $\sum_t \nabla \tilde{\mathcal{L}}_t$, where $\tilde{\mathcal{L}}_t$ is $\mathcal{L}_t$ divided by a running estimate of its own magnitude. So the router never sees $w$.

What Exists and What Is Missing

Three lines of work motivate the proposal by showing the pieces exist separately.

The first is local-global expert routing for segmentation. FLoMo-Net soft-routes over seven multi-kernel local experts plus a global branch and reports 96.02% mDSC on BUSI. Mamba-HoME arranges soft experts into local-group and global levels and reports 78.2% mDSC on PANORAMA against a prior best of 76.6%. Both train with fixed sums of losses. Mamba-HoME also depends on large-scale pre-training followed by fine-tuning. We reuse their local/global expert design but change how the losses reach the router and the experts.

The second is dual-scale routing. GLIDER shows that purely token-level routers switch experts erratically, and fixes this by adding a global task-level routing signal, but only for text and only over pre-trained, frozen experts. We borrow the idea of a task-level routing anchor for dense images, but train it end to end.

The third is task routing combined with loss weighting. Denoising Task Routing shows that hand-crafted channel routing and loss weighting are complementary in diffusion models. This constrains the proposal: routing alone will not remove balancing sensitivity, so we do not claim that it does. Our claim is narrower and testable. The way balancing interacts with a learned router is the problem.

The optimizer-side work supplies the method. CAGrad maximizes worst-case task improvement near the average gradient. LDC-MTL gives a first-order bilevel balancer that costs $O(1)$ in task count. Both assume a fixed shared trunk. Neither distinguishes router parameters from expert parameters, and neither can tell that an expert used by one task needs no balancing.

On the multi-stage side, IMTL collapsed a two-stage speech translation pipeline into one stage with dynamic task weighting, reaching SpeechUT's performance in 20.8% of the training time. AlignDet aligned pre-training with detection fine-tuning. IMTL uses a fully shared network. We ask whether the same single-stage collapse works when the model is routed and the instability comes partly from the router.

One tension must be faced head on. Work on decoupled training, such as Decoupling Representation and Classifier for Long-Tailed Recognition, argues that separate stages protect representations from dominant data. Our single-stage experiment is designed so that this view can win. If it does, that is a reportable result.

Research Program

Model. We build a 2D U-shaped segmentation network. Its encoder blocks each contain a pool of experts: several local convolutional experts with different kernel sizes and dilations, in the spirit of FLoMo-Net, plus one global attention expert. Each block has a task-conditioned soft router. The router combines a learned per-task embedding (the global anchor, after GLIDER) with a per-token score (the local part). Each dataset is a task with its own segmentation head. The losses per task are Dice plus cross-entropy, plus a shared load-balancing term on the router.

Two gradient paths. Implementing the separation needs two pieces. The first is a backward hook on each gate edge that multiplies the incoming gradient by $w_{t,e}$; this costs nothing extra because each task's gradient already flows separately through its gate. The second is a separate, normalized loss for the router parameters. The $w_{t,e}$ are updated once per step by the discrepancy rule above, using gate masses and losses already computed in the forward pass.

Measuring leakage. We also build a diagnostic for the claimed mechanism. During a loss-weight sweep we record how much router mass shifts toward each task's top experts. We correlate that shift with the loss weight, and with the final per-task accuracy drop. If our hypothesis is wrong, router mass will be insensitive to loss weights in the plain model, and the method's benefit, if any, must come from elsewhere.

Generality check. To check that the mechanism is not specific to medical images, we port the same routed decoder-side expert pool to NYU-v2 (segmentation, depth and surface normals). There, every pixel serves all three tasks through task-specific gates over a shared expert pool.

Baselines

The naive reference is the same routed model trained with uniform loss weights and a fixed load-balancing coefficient. We run it at its best setting and across the full sweep, so both its peak and its fragility are visible.

The second reference is the strongest non-routed hybrid, TransUNet. It reports 77.48% average DSC and 31.69 Hausdorff distance on Synapse, and 89.71% average DSC on ACDC; it is the published number on our core datasets that the file reports in full. On BUSI the reference is FLoMo-NetB2 at 96.02% mDSC. For ACDC, MC and CVC-ClinicDB, the file reports that FLoMo-Net leads but gives no per-dataset number, so no reported result was found on Lacuna for FLoMo-Net's individual scores on those three datasets. We will rerun it with its public settings where possible.

The strongest competitor on the actual question is the identical routed model trained with LDC-MTL, and separately with CAGrad, applied globally to all parameters including the router. These are the most capable existing balancers, and they cost about the same compute. On NYU-v2 the target is CAGrad's reported $\Delta m$ of 0.20% (from the CAGrad paper; lower is better). LDC-MTL's NYU-v2 number is not given in the file, so no reported result was found on Lacuna for LDC-MTL on NYU-v2; we run it ourselves. LDC-MTL's reported CelebA $\Delta m$ of −1.31% sets its credibility as a strong baseline.

All methods share data, architecture, epochs and augmentation. Each gets the same tuning budget: the same number of trials over its own hyperparameters (loss weights, CAGrad's $c$, LDC-MTL's step size).

The ablations isolate the key ingredient. One keeps per-expert balancing but lets the router see the weighted loss, which restores the leak. The other keeps the clean router but uses one global weight per task instead of per-expert weights. Beating the global balancers means two things: equal or better mean Dice or $\Delta m$, and a clearly smaller worst-to-best spread across the weight sweep. A gain in mean accuracy alone would not support the core claim.

Why not just…?

Why not just apply LDC-MTL or CAGrad to the routed model? This is the real threat, and it is a direct arm of the study. A global balancer rescales whole task losses, so the rescaled losses still steer the router, and the leak remains. It also spends balancing effort on experts that only one task uses. If global LDC-MTL matches our method on both peak accuracy and sweep spread, the leakage hypothesis is wrong for this setting, and we will say so. The "leak restored" ablation tells us whether any gap comes from the router separation or from per-expert granularity.

Why not just give each task fully private experts, so there is nothing to balance? Hard task partitioning removes conflict by construction, as structural-routing work such as MiLoRA argues. But it also removes transfer, which matters most for small datasets like BUSI and MC. We include a private-experts arm at matched parameter count. If it matches the shared-pool methods on the small datasets, sharing is not buying anything here, and the proposal reduces to an architecture choice.

Why not just grid-search the loss weights once? With five tasks plus a load-balancing term, a grid is expensive. Moreover, the best weights from one dataset mixture need not carry over to another. The Dosovitskiy et al. experience with composite losses is exactly this trial-and-error. We report the tuned grid-search model as an upper reference and count its total GPU-hours. If it wins only with several times our tuning budget, the comparison favors our method on cost. If it wins at equal budget, our method has no practical case.

Why not just use soft routing, which already avoids expert collapse? Mamba-HoME's soft routing does prevent collapse in the sense of unused experts, at a reported 30% inference slowdown. But a soft router can still shift its mass toward the experts favored by an over-weighted task. Our router is already soft, so any gain we show is on top of soft routing, not a replacement for it.

Proposed Experiments and Decisive Checks

Sensitivity sweep. The core experiment trains on BUSI, ACDC, Synapse (sliced to 2D, following TransUNet), Montgomery County chest X-ray and CVC-ClinicDB jointly. We use a log-spaced sweep over two quantities: one task's loss weight (a factor of 0.1 to 10) and the load-balancing coefficient. Reported metrics are per-dataset Dice, HD95, the worst-to-best Dice spread across the sweep, router entropy, and the leakage correlation.

Single-stage versus two-stage. For each target dataset, we compare one-stage joint training against pre-training on the other four followed by fine-tuning on the target, at matched total GPU-hours. We report accuracy and wall-clock cost, in the manner of IMTL's GPU-day accounting.

Generality. The NYU-v2 port is reported in $\Delta m$, against CAGrad's 0.20%.

Decisive outcomes. Three outcomes would settle the claim. If the plain routed model's router mass does not move with loss weights, the mechanism is refuted. If global LDC-MTL closes the sensitivity gap, the per-expert, router-separated design is unnecessary. If single-stage training falls short of two-stage on the small targets even with our method, the decoupling view in the long-tailed recognition literature holds for routed segmentation models, and that is the finding we report.

An extension to 3D (PANORAMA, against Mamba-HoME's 78.2%) follows only if the 2D results are positive.