Stick-Breaking Gates for Sparse Mixture-of-Experts Density Models

A mixture of experts (MoE) is a model in which a small gating network decides, for each input, how much weight each of several sub-models ("experts") gets. Its output can be a full conditional probability density $p(y|x)$, not just a prediction. MoEs are the workhorse of conditional computation, where each input runs only a few experts so that total parameter count can grow while per-input compute stays flat. The statistical theory of MoEs has found a persistent problem. When a model has more experts than the data needs, the predicted density still converges at the fast $n^{-1/2}$ rate. The experts' own parameters, however, converge far more slowly, sometimes as slowly as $1/\log n$. The cause is the softmax gate: its weights must sum to one, and this couples the gate's parameters to the experts' parameters. Gates without that constraint, such as independent sigmoids, avoid the problem, but they do not produce a valid density.

We propose a gate that keeps exact normalization without softmax's competition between experts: a stick-breaking gate. Expert $k$ takes a sigmoid-controlled fraction of whatever probability mass the experts before it left over. This construction has three useful consequences. Every factor in the gate is a sigmoid, so we expect it to escape the parameter-coupling trap the way sigmoid gating does for regression. The ordering breaks the symmetry under relabeling experts, and it pushes unneeded experts to the end of the order, where they receive little mass. Finally, the leftover mass gives an exact, per-input stopping rule for how many experts to run. Under that rule the density error is bounded by twice the skipped mass, so statistical redundancy and skipped compute become the same thing.

Core Claim

We define the radial stick-breaking gate as

$$g_k(x) = \sigma\big(a_k(x)\big)\prod_{j<k}\big(1-\sigma(a_j(x))\big), \qquad a_k(x) = b_k - \frac{\lVert x-c_k\rVert^2}{s_k},$$

where $\sigma$ is the logistic sigmoid and $c_k, s_k, b_k$ are expert $k$'s gate center, width and offset. The last expert receives the remaining mass, so $\sum_k g_k(x)=1$ for every input $x$. The model is $p(y|x)=\sum_k g_k(x) f_k(y|x)$ with Gaussian experts $f_k$.

The claim has three parts. First, this family is dense in $L_p$ for conditional densities on compact domains, with or without truncation. Second, in the overspecified regime its parameters escape the softmax coupling trap, so gate and expert parameters converge at polynomial rates rather than at $1/\log n$. Third, mass-based truncation lets us run only the experts needed for each input while controlling density error exactly. Truncation means stopping once the remaining stick mass $r_K(x)=\prod_{j\le K}(1-\sigma(a_j(x)))$ falls below $\varepsilon$, and assigning that remainder to a cheap broad default density $f_0$. The truncated model $\tilde p$ satisfies

$$\lVert p(\cdot|x)-\tilde p(\cdot|x)\rVert_1 \le 2\,r_{K(x)}(x)\le 2\varepsilon,$$

which follows directly from the triangle inequality.

The second part is a conjecture to be proved or refuted. The first and third parts are the parts we expect to establish cleanly.

Position Relative to Existing Work

The motivating evidence comes from four papers that study gating choices. Each one leaves a different gap.

These two activation-based fixes are the real competitors, because they remain normalized. Our design differs in one structural way: normalization comes from sequential leftover mass rather than a shared denominator. We test whether that matters.

On universality, Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models proves that dense softmax and Gaussian-gated MoEs are universal on compact domains. Its proof has three steps: build a fine partition, approximate the partition indicators with smooth gates, then fill each cell with a location-scale mixture. We reuse this proof structure. The new step is showing that ordered radial sigmoids can approximate any fine partition. Take a finite cover of $[0,1]^d$ by balls of radius $\delta$, in any order. The cells "ball $k$ minus earlier balls" all have diameter at most $2\delta$ and together cover the cube. A steep sigmoid of a radial logit approximates each ball's indicator, and products of $(1-\sigma)$ factors approximate the "minus earlier balls" part.

With linear logits, a stick-breaking gate can only peel off half-spaces from a convex remainder. We expect that version to be strictly weaker, and we will say so rather than claim it.

On redundancy, two lines of work remove extra experts after training. Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency without Model Sweeps merges redundant experts after training to restore $n^{-1/2}$ rates. Factorized Asymptotic Bayesian Inference for Mixture Modeling shrinks redundant mixture weights during training, but only for unconditional mixtures. Both treat redundancy as something to clean up. We ask instead whether the gate itself can place redundancy where compute is never spent.

Singular learning theory, as in Algebraic Information Geometry for Learning Machines with Singularities and Deep Learning is Singular, and That's Good, serves as a diagnostic lens. A singularity is a set of parameter values where several settings give the same model. The theory says the learning coefficient $\lambda$, a measure of effective dimension, governs density-level generalization. We will check whether gate choice changes $\lambda$ while changing parameter rates. That would confirm that the two quantities measure different things.

Research Program

The theoretical work has two parts. The first is the universality theorem for the radial stick-breaking MoE, in dense form and in truncated form, on $[0,1]^d\times\mathbb{R}^q$. The truncated form follows from the dense one plus the $2\varepsilon$ bound. We will also show that, with steep gates, a top-1 rule loses only the mass near cell boundaries, which shrinks in measure. This gives an $L_p$ answer to the open question of whether hard sparse routing preserves universality, at least for this gate.

The second part is a convergence analysis of maximum likelihood in the exact-fitted and overspecified regimes. We adapt the Voronoi loss of the sigmoid and top-$k$ papers, which measures parameter error by matching each true expert to the fitted experts nearest it. The key computation is easy to state: the derivative of $g_k$ with respect to its center equals $g_k(1-\sigma_k)\cdot 2(x-c_k)/s_k$. The extra $(1-\sigma_k)$ factor, absent in softmax, is what broke the coupling equations for sigmoid regression. Whether it also breaks them alongside the Gaussian expert's location and scale derivatives is the question we must settle. If it does not, we will characterize the residual slow directions.

The algorithmic work is a training procedure for stick-breaking MoEs. Its main pieces are:

  • the mass-truncated forward pass;
  • a check for ordering bias, since early experts may absorb too much mass; we treat this as a measured risk and compare against randomized-order restarts;
  • a stick-tail pruning rule: drop experts whose average stick mass falls below $\varepsilon$, which yields an expert count without a sweep over model sizes.

Baselines

The naive reference point is a dense softmax-gated Gaussian MoE trained by maximum likelihood with the same number of experts. It shows what the field does by default. Its sparse sibling, the top-$k$ softmax MoE of Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts, is the standard conditional-computation model. We beat it if our overspecified parameter-error slope is clearly steeper at equal density likelihood.

The strongest existing method is the activation dense-to-sparse gate of Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?, with top-$k$ or mass-based truncation added. It is normalized and already escapes the coupling trap, so it attacks exactly our question. That paper reports its results only qualitatively, as error-versus-$n$ curves matching $n^{-1/2}$ (exact-fitted) and $n^{-1/4}$ (overspecified). No numerical slope is reported on Lacuna for this suite. We will therefore rerun it ourselves.

For regression, the closest reported numbers come from Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts on its 32-dimensional synthetic suite:

  • with ReLU experts, Voronoi loss decays at $n^{-0.51}$ for sigmoid gating versus $n^{-0.24}$ for softmax;
  • with linear experts, the rates are $n^{-0.40}$ versus $n^{-0.11}$.

Normalized sigmoid gating ($\sigma_i/\sum_j\sigma_j$) is a further baseline. It tests whether simply renormalizing sigmoids revives the trap.

The post-hoc competitor is softmax training followed by dendrogram merging (Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency without Model Sweeps). On the maize proteomics data it selected 2 experts, where AIC selected 18.

For conditional density quality beyond MoEs, we use the Neural Inverse Transform Sampler (Neural Inverse Transform Sampler) as an expert-free reference. It reports state-of-the-art results on four of five UCI datasets, but the file gives no per-dataset numbers.

The ablation that isolates our key ingredient keeps the radial logits $a_k(x)$ identical and swaps only how they are normalized, softmax versus stick-breaking. A second ablation varies the expert order (fixed versus randomized restarts).

All models share the same experts, initialization family, optimizer, number of restarts, and hyperparameter search budget. We also apply the same mass-based truncation rule to every gate, because the $2\varepsilon$ bound holds for any normalized mixture. This keeps truncation from being credited to our design.

Why not just…?

Why not just use the activation-inside-softmax gate with top-$k$? This is the serious threat. It is normalized and already reaches $n^{-1/4}$ in the overspecified case (Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?). What it does not do is place redundant experts in a predictable, never-evaluated region of compute, and its universality under sparsity is unproven. The swap-only ablation settles the statistical question. If its slopes and pruning behavior match ours, the stick-breaking contribution reduces to the universality theorem and the ordering-based pruning, and we will report it that way.

Why not just renormalize sigmoid gates? Dividing by the sum reinstates a shared denominator. That denominator is the structure behind the softmax coupling. Normalized sigmoid is a direct baseline in the synthetic suites, and its overspecified slope answers the question.

Why not just train softmax and merge afterwards with dendrograms? Merging restores $n^{-1/2}$ rates for linear Gaussian MoEs. It does nothing for training-time compute, and its guarantees do not cover sparse routing. We run it as a baseline on the same suites and on the maize data. If merged softmax matches stick-breaking on both parameter error and selected expert count, the in-gate approach has to justify itself on compute alone.

Why not just accept slow parameter rates, since the density converges fast anyway? The density rate is indeed unharmed in Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts. The costs show up elsewhere. Slow parameters mean unstable expert assignments. They also mean that unpruned experts cost real compute. And Singularity, misspecification and the convergence rate of EM shows that singular fits slow optimization itself, which matters at finite budgets. We measure all three: assignment stability across seeds, experts evaluated per input, and iterations needed to reach a target likelihood.

Proposed Experiments and Decisive Checks

Synthetic recovery.

The decisive check is the log-log slope of Voronoi loss in the overspecified regime. Stick-breaking should be clearly steeper than standard softmax and at least as steep as the activation gate, at matched held-out log-likelihood. We report slopes separately for exact-fitted and redundant experts, since heterogeneous rates are expected. A slope near softmax's would refute the conjecture.

Learning-coefficient diagnostic. On the smallest synthetic models, we estimate the learning coefficient by sampling from the posterior, in the style of Deep Learning is Singular, and That's Good, for each gate. The prediction is that density-level $\lambda$ barely changes across gates while parameter slopes differ.

Real data under misspecification.

  • On the maize proteomics data, we compare the stick-tail pruned expert count against DSC's 2 and AIC's 18.
  • On the UCI GAS and HEPMASS sets, we build conditional tasks by predicting half the features from the rest. We report held-out conditional negative log-likelihood against average experts evaluated per input, as $\varepsilon$ is swept for every gate.

Our design wins this test if its likelihood-versus-compute curve dominates, rather than only one point on it.

Stress test. As an optional larger-scale check, we use stick-breaking MoEs as the per-pixel output distribution of an autoregressive CIFAR-10 model, measuring bits per dimension against experts evaluated. NITS-CONV's reported 2.97 bits/dim (Neural Inverse Transform Sampler) is a reference for the regime, not a matched comparison, because the backbones differ.