Balancing Experts by What They Change, Not How Often They Fire

Sparse mixture-of-experts models split one large layer into many smaller sub-networks ("experts") and let a small gating network, the router, send each input to one or a few of them. Left alone, the router tends to collapse: a few experts get most of the inputs, improve faster, and attract even more, while the rest never train. The standard fix is a load-balancing penalty that pushes every expert toward an equal share of inputs. In clinical and sensor time series this fix is aimed at the wrong quantity. The moments that matter, such as a patient sliding into sepsis, are rare and bursty. An expert that specializes in them should sit mostly idle. A balance penalty counts that idleness as death and fills the expert with ordinary steady-state data until it stops being a specialist.

We propose to measure an expert's usefulness by how much of the model's change in prediction it accounts for, rather than by how many inputs it receives. The idea comes from dynamic feature importance, the line of work that explains why a time-series model's output moved between step $t-1$ and step $t$. In a sparse model, the change in a layer's output splits naturally across the experts that were active at the two steps. We use that per-expert share of prediction change in two places. During training, it separates dead experts (few inputs, no influence) from dormant specialists (few inputs, large influence during transitions), and only the first group is pushed to take more load. At inference, the same quantity is a by-product of the forward pass and tells a clinician which expert, and through it which inputs, drove a sudden rise in risk.

Core Claim

Prediction changes in monitoring tasks are concentrated in a small fraction of time steps. So an expert that handles 2% of tokens can carry a large share of the total absolute prediction change, while an expert handling 2% of tokens because it is starving carries almost none. A balancing objective with a floor on each expert's share of prediction-change attribution, plus only an upper cap on token load for hardware reasons, should keep rare-regime experts specialized without leaving dead experts. Token-count balancing cannot make this distinction, whether done per batch or over the whole population.

Concretely, let $h_t$ be the input to a sparse expert layer at step $t$, $g_e(h_t)$ the router weight of expert $e$ (zero if not selected), $f_e$ the expert's output, and $y_t$ the model's scalar prediction. We define the expert's contribution to the prediction change as

$$a_e(t) = \int_0^1 \nabla_{z} y\big(z(\alpha)\big)^{\top} \frac{d}{d\alpha}\Big[g_e(h(\alpha))\, f_e(h(\alpha))\Big]\, d\alpha ,$$

where $h(\alpha)$ moves along the observed trajectory from $h_{t-1}$ to $h_t$ and $z(\alpha)$ is the layer output along that path. Summed over experts, this recovers the part of $y_t - y_{t-1}$ that flows through the layer. It is the same completeness property that Delta-XAI proves for its SWING path integral over input features. In training we approximate the integral with two to four points. The functional load of expert $e$ is an exponential moving average $\ell_e$ of $|a_e(t)|$ normalized by the total over experts. The regularizer is a log-barrier floor,

$$R = -\lambda \sum_{e} \log\big(\ell_e - \epsilon\big)_{+} ,$$

with $\epsilon$ a small floor such as a tenth of uniform share. An expert with small token share but large $\ell_e$ pays nothing. An expert with negligible $\ell_e$ is pushed to receive inputs. The EMA-and-barrier machinery is borrowed from φ-Balancing, which tracks population-level expert load with dual "prices" and an entropy barrier. We change the quantity being balanced, not the optimizer.

Research Program

The work has three parts. The first is a measurement study, which is cheap and could falsify the premise. On MIMIC-III and PhysioNet 2019 we train a standard token-choice sparse temporal Transformer with a conventional balance loss and with no balance loss. We then compute per-expert token share and per-expert share of prediction change. The premise holds if, without balancing, some low-token experts carry a disproportionate share of prediction change during labeled deterioration windows, and if the conventional balance loss erases that concentration. If prediction-change mass turns out to be spread roughly in proportion to token counts, the proposal reduces to ordinary balancing and we stop.

The second part is the training method above. We keep hard top-$k$ sparsity and an upper capacity cap per expert, so the hardware concern raised by BASE Layers is still respected. We replace the lower side of the balance constraint with the functional-load floor. The router itself stays a standard linear gate on the current hidden state. That keeps the contribution to the collapse question separate from routing-architecture changes.

The third part is explanation. Because $a_e(t)$ is computed during the forward pass anyway, the model emits an expert-level account of every jump in risk at almost no extra cost. For feature-level attributions we run SWING only through the one or two experts that dominate $a_e(t)$, instead of through the whole network. We test whether this restricted run matches full SWING faithfulness at lower cost; the paper reports SWING at roughly 0.35 seconds per sample.

This positions the work against prior hybrids in specific ways. MERGE routes multimodal tokens into fixed redundancy, uniqueness and synergy pools using time-lagged information decomposition. It decides routing from input statistics and never studies starvation or balance. We leave routing inputs alone and change what counts as an expert being used. Interpretable Mixture of Experts has a utilization loss and a smoothness loss over time, but uses low-capacity linear or tree experts. Its smoothness term actively discourages the abrupt switches that a rare-event expert needs. Dynamic TMoE handles regime shift by spawning and pruning experts after a drift detector fires. It targets forecasting error and produces no attributions. Fuzzy-MoE explains routing with fuzzy rules but runs all experts densely. The anti-homogenization losses of Advancing Expert Specialization for Better MoE, which add orthogonality between expert outputs and confident routing, motivate our concern that uniform load erodes specialization. They are complementary rather than competing, and we test them in combination.

Baselines

The naive reference is a sparse temporal Transformer with no balancing loss at all. This shows how bad collapse actually is in this domain and gives the dead-expert count to beat. The same backbone, data splits, and hyperparameter search budget (an equal number of trials over learning rate and regularizer weight) are used for every variant.

The conventional baseline is the Switch/ST-MoE-style batch-level auxiliary balance loss, tuned over its coefficient. Beating it means equal or better AUROC overall and clearly better AUROC on deterioration windows, with no more dead experts.

The strongest existing anti-collapse method is φ-Balancing. It removes the mini-batch noise problem and is the most direct competitor, because population-level balancing might already tolerate bursty experts. We implement it on the identical backbone with its EMA and entropy barrier and tune the barrier weight with the same budget. No reported result was found on Lacuna for φ-Balancing on clinical time series; its reported gains are on language models (for example, Math500 accuracy rising from 64.8% to 66.2% on Moonlight-16B-A3B). We also run Expert Choice routing (Mixture-of-Experts with Expert Choice Routing), which balances load by construction.

For task accuracy context on the multimodal benchmarks, the strongest reported number is MERGE's in-hospital mortality AUROC of 85.40 on MIMIC-IV and 91.37% accuracy on PAMAP2 (versus 87.74% for FuseMoE), both from MERGE. We do not need to beat MERGE outright, since its gains come from routing design. The goal is to show that our regularizer added to MERGE's expert pools does not hurt and protects its rarely used synergy pool.

For explanations, the strongest baselines are full SWING via the Delta-XAI codebase and FIT (What went wrong and when?). FIT reports an AUROC of 0.798 for recovering important features on its synthetic State data and a deterioration-test drop of 0.71 on MIMIC mortality. Delta-XAI reports that SWING beats 14 baselines across nine metrics, but per-metric values are not available in the file.

The ablation that isolates the key ingredient keeps everything identical except the balanced quantity. It applies the same barrier to raw output magnitude $\|g_e f_e\|$ instead of the change-attribution $|a_e(t)|$. If this works as well, the temporal-change part of the idea is doing nothing.

Why not just…?

Why not just use φ-Balancing, which already balances over the population rather than the batch? This is the most serious threat. Population-level balancing fixes the case where one batch happens to be all one domain. However, it still targets a roughly uniform long-run token share, so an expert serving a regime that appears in 2% of time steps is still asked to absorb about $1/E$ of all tokens. Whether that pressure actually homogenizes the rare expert in practice is an empirical question. The controlled burst-frequency experiment below answers it directly. If φ-Balancing keeps the rare-regime expert specialized down to low burst frequencies, our contribution shrinks to the explanation side, and we will say so.

Why not just use Expert Choice routing or turn the balance coefficient down? Expert Choice gives every expert a fixed quota per batch, which is exactly the wrong prior when one regime should own a long stretch of time, as Dynamic TMoE argues for non-stationary series. Lowering the coefficient trades homogenization back for dead experts, the loop described in Scaling Vision with Sparse Mixture of Experts. Our sweep reports both dead-expert counts and rare-regime specialization along the full coefficient range. If some coefficient matches our method on both, the alternative wins.

Why not just train a dense model and explain it with SWING or FIT? If explanation quality is all that matters, this is a legitimate choice. It is also the reference against which our feature-level attributions are scored. Our claim is narrower: a sparse model gives expert-level change accounting at no extra cost, and running SWING through only the dominant experts reaches comparable faithfulness at a fraction of the 0.35 seconds per sample. We report both numbers. If the restricted run is not cheaper, or is less faithful, the explanation half of the proposal fails while the training half stands or falls on its own.

Why not just feed the router the input change $x_t - x_{t-1}$? This would let the router see transitions without any new objective. It does not change the balance penalty, however, which would still force the transition expert to take steady-state tokens. We run it both alone and combined with our regularizer, so its effect is separable.

Proposed Experiments

The decisive check is a controlled burst benchmark built from the Switch-Feature and Delayed Spike generators in the Delta-XAI codebase. We plant a rare regime with known ground-truth causal features and vary its frequency from 20% down to 1% of time steps. At each frequency, every balancing method is measured on four things: error inside the rare regime, mutual information between expert identity and regime label, number of dead experts, and the Gini coefficient of token load. The prediction is specific. As frequency falls, token-balanced methods (batch-level, φ-Balancing, Expert Choice) should lose rare-regime specialization and error. Our method should keep it while holding dead experts near zero. The magnitude ablation should fall between them. If the curves coincide, the core claim is wrong.

On real data, we use MIMIC-III and PhysioNet 2019 through the Delta-XAI loaders for online mortality and sepsis prediction. We report AUROC overall and within six hours before onset, along with the same expert-health metrics. Feature-level faithfulness uses Delta-XAI's nine evaluation metrics, comparing restricted SWING against full SWING and FIT, together with wall-clock cost per explained step. On MIMIC-IV and PAMAP2 we add the regularizer to a MERGE-style model and check whether the synergy pool, the one most likely to starve, keeps its specialization without lowering the reported 85.40 AUROC and 91.37% accuracy. A smaller secondary run on ETTh1 and Weather with the Dynamic TMoE codebase tests whether a fixed expert pool with our floor can match spawn-and-prune drift handling without its cold-start error spikes. That paper reports only aggregate relative gains (10.4% lower MSE across nine benchmarks), so we rerun it ourselves under matched budgets.