Routing by Lateral Inhibition: Mixture-of-Experts Gates as Small Threshold-Linear Competitive Networks

A sparse mixture-of-experts model is a network that holds many sub-networks ("experts") but sends each token through only a few of them. A small learned "router" scores the experts and keeps the top few. These models fail in two opposite ways. Without constraints, a few experts win early, receive most of the training signal, win even more, and the rest die. This is expert collapse. The usual fix is an extra loss that pushes the router toward uniform usage, and that causes the opposite failure: experts receive indistinguishable data and learn nearly the same function, so the model behaves like one dense layer. Brains face the same tension between keeping neurons alive and keeping them distinct. They handle it with two separate mechanisms. Lateral inhibition, in which active neurons suppress similar neighbours, keeps neurons distinct. Slow self-adaptation, in which a neuron that fires too often becomes harder to excite, keeps the population in use.

We propose to build the router as exactly that circuit. For each token, the experts compete in a tiny recurrent network with rectified-linear (non-saturating) units. Experts with similar input keys inhibit one another, and each expert carries a firing threshold that rises with its recent load and falls when it is starved. The settled activity of this network is sparse and decides which experts run. The claim is that separating the two jobs fixes the tradeoff. Balance comes from a threshold that never sends gradient into the experts. Specialization comes from inhibition between similar experts. Together they should avoid both collapse and homogenization without an auxiliary balancing loss and without letting tokens compete with each other, which keeps the router valid for left-to-right language modeling.

Core Claim

The research file points to a real contradiction. In competitive-activation work, losing units receive zero gradient, and this is what reserves them for other inputs and drives specialization (Compete to Compute, Winner-Take-All Autoencoders). In mixture-of-experts work, the same zero gradient is identified as the cause of runaway expert death (Scaling Vision with Sparse Mixture of Experts). Current fixes in that literature push the router toward uniform usage, which Advancing Expert Specialization for Better MoE and Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer show homogenizes the experts.

Our claim is that the contradiction comes from using one mechanism for two jobs. Starvation is a population-level problem and should be solved by a non-gradient homeostatic threshold. Redundancy is a pairwise problem and should be solved by learned inhibition between similar experts. The competition needs non-saturating units. Two things follow from that choice. The settled state is exactly sparse rather than a softmax drifting toward one-hot. Winners pass gradient linearly. And because competition happens among experts within a single token, never across tokens, the router stays causal.

Mechanism

For a token with hidden state $x$, the router computes raw affinities $a = K x$. Here $K$ holds one key per expert, $L_2$-normalized in a low-dimensional projection following On the Representation Collapse of Sparse Mixture of Experts. The router then runs $S$ steps (we plan $S$ between 3 and 5) of a threshold-linear competition. It starts from $r^{(0)}=0$ and updates

$$r^{(s+1)} = \mathrm{ReLU}\big(r^{(s)} - \eta\,[(I + W)\,r^{(s)} - (a - \theta)]\big),$$

where the symbols are defined as follows. $r$ is the vector of expert activities. $W = g\,\hat K \hat K^\top$ with the diagonal removed is the inhibition matrix. It is built from the normalized expert keys $\hat K$ and a learned gain $g \ge 0$, so experts with similar keys suppress each other most. $\theta$ is the per-expert threshold, and $\eta$ is a step size.

With symmetric $W$ and $I+W$ positive definite, this update is projected gradient descent on the non-negative quadratic program $\min_{r\ge 0} \tfrac12 r^\top (I+W) r - (a-\theta)^\top r$. It therefore converges for $\eta < 2/\lambda_{\max}(I+W)$, which is the standard step-size bound for gradient descent on a quadratic. Its fixed point is sparse: many experts sit exactly at zero.

Experts with nonzero $r$ run, and their outputs are weighted by $r$. To keep compute bounded we cap the number of active experts at $k_{\max}$ and tune the mean active count to match a top-$k$ baseline. The cost per token is $O(S N^2)$ for $N$ experts, which is negligible next to the expert feed-forward layers at $N \le 64$.

The threshold is a slow state updated outside backpropagation once per training step: $\theta_e \leftarrow \theta_e + \gamma(\ell_e - \bar\ell)$. Here $\ell_e$ is expert $e$'s running share of tokens, $\bar\ell$ is the uniform share, and $\gamma$ is a small rate. A starved expert's threshold falls until it wins the tokens nearest its key. Those tokens give it gradient, so it can specialize instead of being forced onto random data.

Relation to Prior Work

Two lines of work already join routing and competition. They differ from this proposal in where the competition lives.

Sparse Mixture of Experts as Unified Competitive Learning solves a linear program over a batch-level token-by-expert score matrix. It uses sigmoid scores to limit cross-token leakage. It has no expert-to-expert coupling and no settling dynamics. Our router has no cross-token coupling at all; its coupling runs between experts.

Localist Topographic Expert Routing uses lateral inhibition to keep 39 tactile modules distinct, but on a fixed physical neighbour grid with spiking neurons. Our inhibition is learned from expert keys in an abstract semantic space.

The file also contains depth-wise recurrent routing. Layerwise Recurrent Router for Mixture-of-Experts passes routing state between layers through a GRU and reports that standard routers collapse prematurely to near one-hot scores. MomentumSMoE adds momentum to residual updates across layers. Both motivate this proposal, because they show that routing dynamics matter. Both are orthogonal to it, because their recurrence runs across depth while ours is a within-layer settling process.

We deliberately do not pursue the file's "Gap 1", which would replace the RMoE GRU with a non-saturating cell. Depth is 12 to 32 steps, far below the thousands of steps where IndRNN and the norm-stabilizer show saturating cells failing. The expected gain from that swap is therefore small.

Bridging Game Theory and Transformer Routing also reaches load balance endogenously, through a congestion equilibrium. It does so with dense soft routing at a reported 1.65× training-time overhead. Our version replaces congestion with a cheap threshold.

Two papers supply the method. Structured sparse coding via lateral inhibition provides inhibition-driven sparse code selection. Global stability of a Hebbian/anti-Hebbian network provides precedent for stability guarantees in laterally inhibitory networks.

One paper constrains the design. The non-saturating recurrence literature, including Towards Non-Saturating Recurrent Units, reports gradient spikes under unbounded units. This is why we parameterize $W$ so that convergence is guaranteed rather than relying on a norm penalty.

Research Program

We will first implement the router as a drop-in replacement for top-$k$ gating in the medium sparse mixture-of-experts language model used by MomentumSMoE on WikiText-103. That setup gives us a published baseline to reproduce. We will then measure five things through training:

  1. Load per expert, using the coefficient of variation and the count of experts below 1% load.
  2. Expert homogenization, using pairwise similarity of expert outputs on a fixed probe set, measured as linear centered kernel alignment.
  3. Gate sharpness over time, using the entropy of $r$ normalized over active experts, compared against the premature one-hot convergence that RMoE documents.
  4. Routing stability across seeds, which X-MoE also uses.
  5. Token drops, which should be zero by construction because there is no capacity buffer.

We will also test analytically and empirically whether the learned gain $g$ grows to keep similar experts apart. We will check whether expert keys spread out in angle without any explicit orthogonality loss. Finally, we will move to the X-MoE masked-language-model and XTREME setting to check transfer to a second regime with a published baseline.

Baselines

The naive reference is a standard top-2 token-choice sparse model with no balancing loss. It shows how severe collapse is in this setup, and our router must beat it on dead-expert count and perplexity.

The conventional practice is top-2 with a Switch-style auxiliary balancing loss and a capacity factor. It is the most widely deployed fix, so it is the main collapse baseline. We must match or beat it on load balance while showing lower expert-output similarity and lower perplexity. In the MomentumSMoE setup the reported baseline is 33.76 perplexity on WikiText-103.

The strongest reported number on this benchmark in the file is AdamSMoE-medium at 31.59 perplexity on WikiText-103 (MomentumSMoE). AdamSMoE changes the residual stream rather than the router. We will therefore compare against it directly and also stack our router on top of it. Beating it means a lower mean perplexity across three seeds against our own reproduction, with non-overlapping seed ranges.

Two router-level competitors complete the set:

  1. X-MoE cosine routing, which reports 18.72 validation perplexity against 19.02 for the standard sparse baseline on multilingual masked language modeling, and 65.3 against 63.8 on XTREME (On the Representation Collapse of Sparse Mixture of Experts).
  2. RMoE. We found no reported result on Lacuna for RMoE on WikiText-103, so we will run it ourselves.

All methods use the same data, tokenizer, model width and depth, number of experts, training tokens, and mean active experts per token. Each method gets the same hyperparameter sweep budget, a fixed number of configurations over learning rate and its method-specific coefficient, with three seeds at the chosen setting.

Two ablations isolate the key ingredients:

  1. Threshold-only: $W=0$, adaptive $\theta$ kept.
  2. Inhibition-only: $W$ kept, $\theta$ fixed.

A third ablation replaces ReLU with a sigmoid in the same settling loop. It tests whether non-saturation itself matters.

Why not just…?

Why not just use an auxiliary balancing loss? It fixes starvation by pushing gradients that equalize usage, and the file's evidence (Advancing Expert Specialization, Orthogonal Optimizer) is that this homogenizes experts. With capacity buffers it also drops tokens; the mean-field paper reports about 11% dropped. The decisive measurement is expert-output similarity at matched load balance. If the auxiliary loss reaches the same similarity and perplexity, our specialization claim fails.

Why not just use an adaptive per-expert threshold or bias with plain top-$k$, and skip the inhibition network? This is the real threat. A load-driven bias alone may already prevent collapse without gradient interference, and we found no reported result on Lacuna for such a variant in this file. Our threshold-only ablation is exactly this method. If it matches the full router on perplexity and expert similarity, we will report that lateral inhibition adds nothing and that the value lies in non-gradient balancing alone. The inhibition term must earn its place by lowering expert similarity at equal load.

Why not just decorrelate expert keys statically, with cosine routing or an orthogonality penalty? X-MoE and the orthogonal-optimizer work push experts apart through parameter geometry. Static separation, however, does not stop two experts from both firing on the same token whenever their keys are only moderately close. Inhibition acts on activity, token by token. We will compare against X-MoE with an added key-orthogonality penalty tuned on the same budget, and report how often pairs of similar experts are co-active.

Proposed Experiments and Decisive Checks

The main experiment trains each baseline and our router on WikiText-103 at the MomentumSMoE medium scale, then on the X-MoE multilingual masked-language-model and XTREME setting. We report perplexity, XTREME average, dead-expert count, load coefficient of variation, expert-output similarity, gate entropy curves, and wall-clock time per 1,000 steps. RMoE reports its own overhead as 972.9 against 960.2 seconds; we will report ours on the same basis. We also include the WikiText-103 word-swap attack from MomentumSMoE as a robustness check.

The proposal is supported only if three things hold:

  1. The full router beats both ablations on expert similarity at equal load.
  2. It is at least as good as the auxiliary-loss baseline on perplexity.
  3. The sigmoid version reproduces premature one-hot gating while the ReLU version does not.

If the threshold-only ablation matches the full router, the lateral-inhibition claim is falsified, and we will say so.