Letting Expert Counts Emerge from Niche Competition During Upcycling

A mixture-of-experts model (a network whose feed-forward blocks are split into many "experts," with a small learned router sending each token to only a few of them) gets more parameters at roughly the same compute per token. Its builders still have to guess two numbers by hand: how many experts each layer gets, and how those experts start out. A common cheap recipe is upcycling, which converts an already-trained dense model into a mixture-of-experts by copying its feed-forward layer into every expert. The copies are identical, so they never learn to differ. Drop-Upcycling fixes this by randomly re-initializing half of each copy's hidden neurons. It uses the same expert count and the same random 50% everywhere, and the authors report that performance is brittle around that ratio.

We propose to choose both the per-layer expert count and each expert's re-initialization mask with one population-level competition held during a short warm-start window. The idea is borrowed from evolutionary model merging. Candidate experts compete for tokens. Each token is a fixed reward split among the experts that serve it well, so a redundant expert earns little and is culled, while an expert that covers tokens nobody else handles survives. The number of surviving experts in a layer becomes that layer's expert count. The surviving experts' neuron masks become their initialization. After the window the topology is frozen and training continues with ordinary static-shape kernels.

Core Claim

Our hypothesis is that implicit fitness sharing over calibration tokens, measured by actual loss, picks a better layer-wise expert-count profile and better re-initialization masks than either uniform hand-set values or activation-based heuristics, at matched total compute. The novelty is concentrated in one ingredient: expert count is defined as the number of niches the token distribution supports, scored by shared next-token loss, instead of being a hyperparameter or a local threshold decision. If the sharing term is removed, or replaced by clustering of activations, the gains should vanish. The ablations below are built to show exactly that.

Where This Sits

Three bodies of work motivate the proposal.

Drop-Upcycling supplies the base method. We reuse its statistics-matched Gaussian re-initialization of dropped neurons, but we replace its random mask and fixed $N$ with searched ones.

Dynamic Mixture of Experts (DynMoE) supplies the motivating evidence. It adds an expert whenever tokens fail every expert's trainable threshold and prunes unused experts, and it reports roughly 15% fewer activated parameters than fixed MoE-LLaVA. It also observes that top layers collapse toward a single expert, which means a uniform $N$ wastes capacity. Its decisions are made from single training steps, though, and dynamic shapes lack fast kernels. We make the count decision once, early, from a population signal, and then go static. The systems cost of dynamic shapes is also stressed by Compute-Optimal Is Not Cluster-Optimal.

Competition and Attraction Improve Model Fusion (M2N2) supplies the selection mechanism. It shares each data point's reward among the population members that solve it, and it pairs parents with complementary errors. It reports 63.49 versus 49.06 for CMA-ES on combined GSM8K/WebShop merging. M2N2 fuses models into one dense network. We instead keep the population members as separate, conditionally activated experts, so the niches themselves become the architecture.

Two closely related works constrain the design. ACO-MoE-LoRA runs an ant-colony controller over LoRA ranks and expert sparsity inside a frozen SAM2 backbone. It is the nearest "evolve-while-training" precedent, but it operates on adapters and requires delicate tuning of the controller. Evolutionary Soups evolves only small gating networks over a fixed set of frozen experts. Neither decides how many full feed-forward experts a layer should have, or how they are initialized.

The search space is shaped by two further cautions. The CMA-ES weakness in high dimensions reported in EvoGrad, which solves 61.2% of COCO functions versus 94.9% for its hybrid, rules out evolving raw weights. We therefore evolve only binary neuron masks and integer counts. The consistency guarantees in Dendrograms of Mixing Measures recover the true expert count without model sweeps, but only for linear softmax gates over Gaussian experts. We take its merge-redundant-experts intuition as inspiration, not as a usable tool for Transformers.

Research Program

Setup. Start from a pretrained dense Transformer. In each layer $l$, instantiate an over-provisioned pool of $N_{\max}$ candidate experts. Each candidate $e$ is described by a mask $m_e \in \{0,1\}^{d_{ff}}$ over the dense layer's hidden neurons: kept neurons are copied, and dropped neurons are re-initialized as in Drop-Upcycling. The population is seeded with random masks at varied drop ratios. The pool is trained jointly with a standard router for a warm-start window of about 5% of the planned upcycling tokens.

Scoring. At intervals during the window, we score candidates on a held-out calibration set of tokens. For token $t$ and layer $l$, we force-route $t$ to candidate $e$ and record the next-token loss $\ell_e(t)$. This costs one partial forward pass per candidate from layer $l$ upward, which is affordable at the model scales below. A candidate "serves" $t$ if $\ell_e(t) \le \ell^*(t) + \delta$, where $\ell^*(t)$ is the best loss any candidate achieves on $t$. Shared fitness is

$$F_e = \sum_t \frac{\mathbb{1}[e \text{ serves } t]}{\sum_{e'} \mathbb{1}[e' \text{ serves } t]},$$

so each token contributes a total reward of one, divided among the experts that serve it. Here $\delta$ is a tolerance hyperparameter. An expert that duplicates others' coverage sees its $F_e$ shrink in proportion to the number of duplicates.

Selection and variation. Candidates whose $F_e$ falls below a floor are culled. The floor is set at a fixed fraction of the tokens an expert would receive under perfectly balanced routing. New candidates come from two sources. Crossover takes masks from parent pairs whose served-token sets overlap least, which is M2N2's attraction rule. Mutation flips which neurons are re-initialized. At the end of the window, the number of survivors is $N_l$, and the survivors' masks and partially trained weights become the expert initialization. The router's top-$k$ is kept fixed, so the comparison isolates the effect of count and initialization.

Cost accounting. Every forced-routing pass is counted in FLOPs and charged against the method's training budget.

Analysis. Beyond accuracy, we will examine three things. First, whether the emergent profile $N_1,\dots,N_L$ reproduces DynMoE's observation of shrinking counts in upper layers. Second, how pairwise expert weight similarity evolves compared with uniform Drop-Upcycling. Third, whether surviving masks keep systematically different neurons across layers, which would explain the brittleness Drop-Upcycling reports around $r = 0.5$.

Baselines

The naive reference point is plain upcycling with identical copies at a fixed $N$, together with the dense model trained for the same extra tokens. Every method must beat the dense model to justify sparsity at all. On the training corpus we use, the file reports a FineWeb-Edu 1.7B dense model at 37% MMLU and 57% ARC (The FineWeb Datasets). That number anchors scale but is not a comparison point at our model sizes.

The strongest existing method is Drop-Upcycling with uniform $r$ and uniform $N$. To keep the comparison fair, we give it the entire compute our search consumes, spent on a grid over $N \in \{4, 8, 16\}$ and $r \in \{0.25, 0.5, 0.75\}$ with short proxy runs, followed by full training of the best cell. The only reported result found in the file is that a 5.9B-active Drop-Upcycled MoE matches a 13B dense model at about 25% of training FLOPs, on the LLM-jp evaluation suite. We found no reported per-benchmark score on Lacuna for Drop-Upcycling at small scale on FineWeb-Edu, so we will reproduce it. Beating it means lower validation loss and higher average downstream accuracy at equal total FLOPs, search included, with the gap exceeding the seed-to-seed spread across three seeds.

The second strong competitor is DynMoE, started from the same Drop-Upcycled pool and given the same token budget. It is the existing answer to "let counts emerge." Its reported advantage is about 15% fewer activated parameters at matched accuracy, with no file-reported numbers at our scale. Beating it means matching or exceeding its accuracy with comparable activated parameters, while also achieving higher throughput thanks to the static post-window topology.

Three ablations isolate the key ingredient. The first keeps the evolutionary loop but removes fitness sharing, so the score is raw $\sum_t \mathbb{1}[e \text{ serves } t]$. The second replaces evolution with random mask search under the same evaluation budget. The third derives masks and counts from activation clustering, described below.

Why not just…?

Why not just grid-search a uniform $(N, r)$ with cheap proxy runs? This is the most likely real threat, because the search space is small and proxy runs are cheap. It is precisely the matched-compute Drop-Upcycling baseline above. The case against it is that DynMoE's top-layer collapse and ACO-MoE-LoRA's finding of very different needs across encoder and decoder layers both imply that the best configuration is not uniform, and a uniform grid cannot represent that. If the grid ties our method, we will conclude that layer-wise heterogeneity matters little at this scale, and we will report that outcome.

Why not just let DynMoE add and prune experts during training? DynMoE spawns experts from tokens unmatched at a single step and needs dynamic shapes throughout training, which brings the kernel penalty noted by both DynMoE and Compute-Optimal Is Not Cluster-Optimal. Our method decides once, from loss aggregated over a calibration set, and then trains statically. The head-to-head run measures both accuracy and tokens per second. If DynMoE wins on accuracy, our method's value reduces to throughput, and we will say so.

Why not just cluster each layer's activations, pick $N$ by a cluster-quality score, and keep each cluster's most active neurons? This is cheaper and needs no population, and it could plausibly work just as well. For that reason we treat it as both a baseline and the seed for one variant of our population. If evolution started from clustering seeds fails to beat clustering alone, the evolutionary component is unnecessary and the contribution shrinks to the loss-based sharing score. The argument for loss over activations is that activation similarity ignores how a dropped neuron changes the layer's output. That argument is the hypothesis being tested, not an established fact.

Proposed Experiments

Primary setting. Dense decoder models at roughly 150M and 600M parameters, pretrained on FineWeb-Edu, then upcycled to top-2 mixtures-of-experts with $N_{\max} = 32$ and trained on further FineWeb-Edu tokens. We report validation loss, HellaSwag, ARC-Challenge, MMLU, and GSM8K, which overlaps the suites used by Drop-Upcycling and the expert-router coupling loss. We also report activated parameters and measured throughput.

Secondary setting. A ViT upcycled on DomainBed, which is DynMoE's own benchmark. This tests whether the count-selection rule transfers to a domain-shift setting where the best count is known to vary.

Decisive checks. First, at matched total FLOPs, our method must beat the tuned uniform grid and DynMoE at both model sizes. Second, removing fitness sharing must raise inter-expert similarity and erase most of the gain. Without that, the niche mechanism is not doing the work. Third, evolution must beat clustering seeds and random search under equal evaluation budgets. Fourth, transfer: the profile found at 150M, rescaled to the 600M model, should keep most of its advantage. If it does, the search cost can be paid once at small scale, which is what makes the method practical. Failure on the first or third check would falsify the core claim.