A mixture-of-experts (MoE) model replaces each feed-forward block with many smaller sub-networks, the "experts." A small learned "router" sends each token to a few of them, so total parameters grow while compute per token stays fixed. Modern MoEs favor many small experts. OLMoE, for example, activates 8 of 64 per token. The router's 64-way decision is made from random initialization, and it locks in almost immediately: OLMoE: Open Mixture-of-Experts Language Models reports routing saturating within the first 1% of training, together with persistent "superstar" experts that absorb most tokens. Graph embedding faced a structurally similar problem a decade ago. Local optimization from random starts got stuck before it found global structure. The fix, in HARP: Hierarchical Representation Learning for Networks and A Fast Multi-Scale Method for Drawing Large Graphs, was coarse-to-fine: solve a small, coarsened version of the problem first, then use that solution to warm-start the full one.
We propose doing the same for routing granularity. Training starts with a few wide experts and top-1 routing, which is an easy 8-way decision. At scheduled points, each wide expert is split into several narrower children by partitioning its hidden neurons. Nothing is copied. The children inherit their parent's routing, with gate values set so the network computes exactly the same function at the moment of the split. Because the children own disjoint neurons, they are never clones. This sidesteps the symmetry problem that copy-based growth must fight with noise or re-initialization. Active and total parameter counts stay fixed throughout. Only the router's freedom grows. The question is whether learning the coarse routing structure first, then refining it, yields better-specialized and better-balanced fine-grained MoEs than training the fine-grained router from scratch for the same number of tokens.
Core Claim
Take a feed-forward expert with intermediate width $d$ that receives gate value $p$ for a token. It can be rewritten exactly as $m$ experts of width $d/m$, one per block of a partition of its neurons. Each child gets gate logit equal to the parent's logit minus $\log m$. Each child's output projection is multiplied by $m$. Under this rewriting, top-1 routing over $E$ parents becomes top-$m$ routing over $mE$ children: the top parent's $m$ children tie for the highest logits and are all selected. The layer output is unchanged for every token.
We call this function-preserving operation an expert split. Our claim is that a schedule of splits, for example 8 experts with top-1 routing, then 32 with top-4, then 64 with top-8, yields a final 64-expert model with lower perplexity, lower inter-expert weight similarity, and less superstar concentration than an identically shaped model trained directly at 64/top-8 on the same tokens.
A secondary claim concerns how neurons are grouped. Partitioning a parent's neurons along communities in their co-activation graph should produce children that specialize faster than a random partition. This is a direct transfer of the finding in Learning Topological Representation for Networks via Hierarchical Sampling, which coarsens graphs by modularity communities and found that arbitrary collapsing damages community structure. We treat this secondary claim as genuinely uncertain and test it as an ablation.
Relation to Prior Work
Two existing lines of work build MoEs by copying. Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization copies a dense FFN into every expert, then re-initializes a random 50% of intermediate dimensions using moment-matched noise. Its own observation is that unconstrained noise destroys pretrained ability while too little leaves the copies collapsed. SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning widens dense models and OLMoE mid-training. It needs two separate fixes: rescaling to avoid loss spikes ("signal shock"), and asymmetric optimizer resets and learning-rate re-warmup to stop copied weights from moving in lockstep ("symmetry lock").
Both papers supply our motivation: symmetry and signal shock are real costs. Both also constrain our design. Splitting avoids these costs by construction. It is exact, so there is no shock. Children share no weights, so there is no lock. Every existing neuron keeps its own optimizer state, and only the new router rows start fresh. The price is that splitting adds no parameters. This proposal studies granularity growth, not capacity growth, which SPARKLING leaves open as part of its missing "expert-granularity" story.
The method closest to our split is ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns. It partitions a trained dense model's neurons into experts in one shot, using balanced K-means on activation patterns from an external calibration set, and initializes router centroids from the GLU gate weights. It then needs 200B tokens of continued pretraining to recover. We reuse two of its techniques: balanced clustering of neurons, and gate-weight-derived router directions for breaking ties between children. We differ in three ways. Our split is exactly function-preserving, because routing is inherited rather than re-derived. It is repeated across stages inside one pretraining run. Its calibration data is the training stream itself.
Hierarchical Routing Mixture of Experts grows routing trees by expectation-maximization on small tabular regressions. Hierarchical Mixture-of-Experts with Two-Stage Optimization imposes a fixed hardware grouping and adds regularizers to keep experts within a group from becoming clones. Neither grows the hierarchy during deep-network training. Our split produces a hierarchy as a by-product: children of one parent form a group. This makes the hardware-aligned placement of each parent's children on one device a natural option. Unlike Hi-MoE, such a placement would not start from clones.
SD-MoE: Spectral Decomposition for Effective Expert Specialization diagnoses collapse as shared dominant singular directions, with inter-expert similarity around 0.7. It fixes this by projecting gradients during training. We borrow its similarity metric as a diagnostic. Its method is orthogonal to ours and could be stacked on top.
One prior paper cuts against us. HSRL notes that HARP-style warm starts can lose the coarse structure as fine training proceeds. If that happens here, the coarse stage would buy nothing. We address it directly with a variant that keeps a parent-level logit added to each child's logit after the split, described below.
Research Program
The base architecture is an OLMoE-style decoder with SwiGLU experts. The target configuration is 64 experts with top-8 routing. The coarse configurations are 8 experts with top-1 and 32 experts with top-4. All three have identical active and total parameter counts, which lets us compare at equal FLOPs, memory, and tokens.
At a split, we take each parent expert and compute co-activation statistics over its neurons on roughly a few million tokens from the current training batch stream. The statistic is the correlation of gated activations among tokens routed to that parent. We then partition the parent's neurons into $m$ equal-size blocks using balanced K-means on activation profiles, following ExpertWeaver. A modularity-based balanced partition is an alternative.
Router rows for the children are set to the parent's row, with a $-\log m$ logit offset. A small tie-breaking term is added: each child's direction is set to the mean of that child's gate-projection rows, scaled by a coefficient $\epsilon$. This is ExpertWeaver's observation that gate weights act as routing centroids. When $\epsilon=0$ the split is exact. When $\epsilon$ is small, the loss is perturbed only to first order.
New router rows get fresh optimizer state and a short learning-rate re-warmup, following SPARKLING's asymmetric treatment. Expert weights keep their state, since they are the same parameters.
For the "hierarchy retention" variant, each child's logit after the split is the sum of a frozen-then-unfrozen parent logit and a child logit. This preserves the coarse structure that HSRL warns can erode.
The split schedule is the main open design variable. We will compare three options: fixed token fractions (split at 2% and 10% of tokens), a criterion-triggered schedule that splits when the coarse router's token-to-expert assignment stops changing (the saturation signal OLMoE measured), and a single split from 8 to 64. Beyond end-task quality, we will analyze several things across training: inter-expert cosine similarity and top-singular-direction overlap using SD-MoE's protocol, the token share of the top expert per layer, routing churn, and whether children of the same parent stay functionally related or drift apart.
Baselines
The naive reference point is direct fine-grained training: the identical 64-expert, top-8 model trained from random initialization with the same load-balancing loss, data, token budget, and learning-rate schedule. This is how OLMoE itself was trained. It is the question's real null hypothesis, since our method changes only the order in which routing freedom is granted. A win means lower validation perplexity at the full token budget and higher average downstream accuracy, with gaps larger than the seed-to-seed spread from three seeds at the smallest scale. Matching its perplexity with lower superstar concentration would count as a partial result, not a win.
The strongest competitor for growing a fine-grained MoE from a coarser one is copy-based expansion. We will implement it as Drop-Upcycling applied to the coarse 8-expert checkpoint. Each parent's width-$d$ expert is shrunk to width $d/m$ by dropping neurons, then copied $m$ times with 50% moment-matched re-initialization. SPARKLING's rescaling and asymmetric optimizer reset are added. The split happens at the same token count as ours, and the total budget is identical. This baseline isolates "split" versus "copy" while holding the coarse stage fixed. We give it the same hyperparameter sweep as our method: drop ratio and re-warmup length, five settings each.
For reference numbers, SPARKLING reports a downstream average of 62.55 against 60.89 for its baseline on ARC, MMLU, SciQ, HellaSwag, and PIQA. SD-MoE reports roughly a 3% average accuracy gain and similarity falling from about 0.7 to below 0.1 on a DCLM 100B-token run. Hi-MoE reports a 5.6% average perplexity reduction on OLMoE-7B with 58B Dolma tokens. These come from different scales and data, so they set the expected size of improvements in this area rather than targets we can match. No reported result was found on Lacuna for any granularity-progressive or split-based MoE in our configuration.
A one-shot downcycling baseline in the style of ExpertWeaver trains a dense model with matching active compute for the coarse-stage token count, then partitions it once into 64 experts and trains the remainder. This tests whether the intermediate coarse MoE stage matters, or whether dense-then-split is just as good. ExpertWeaver reports an average score of 60.4 for Llama3-8B at 25% sparsity, against 57.2 for CMoE. That is a post-hoc setting, not comparable to ours in absolute terms.
The key-ingredient ablations are three. Random-partition splits keep exact function preservation but discard neuron structure. Splits with $\epsilon=0$ remove the gate-weight tie-breaking. The hierarchy-retention router is compared on and off.
Why not just…?
Why not just train the 64-expert model from scratch with a good load-balancing loss? This is the main threat, because it costs nothing extra. Load balancing equalizes traffic but does not make routing meaningful. Improving Expert Specialization in Mixture of Experts shows balanced-but-unspecialized gating on small vision tasks, and SD-MoE measures about 0.7 inter-expert similarity in from-scratch language MoEs. Neither result shows that a coarse start fixes this. The direct-training baseline at three seeds settles it. If the gap is within noise, the proposal fails on its central claim, and we will report that.
Why not just copy coarse experts and perturb them, as Drop-Upcycling and SPARKLING do? Copying needs a carefully tuned amount of noise. Drop-Upcycling found too much noise erases learned function, and SPARKLING needed optimizer resets to escape lockstep updates. Splitting needs neither. It also discards nothing, whereas copy-based shrinking drops half of each child's inherited neurons. The matched copy-versus-split comparison above tests whether this cleaner starting point translates into better final models, or whether copying with perturbations catches up within a few billion tokens.
Why not just split at random, since exact function preservation may be what matters? This alternative may well be as good. Neuron co-activation within one expert could be weak or unstable early in training. If random partitions match structured ones, we will drop the clustering step and present the method as structure-free splitting, which is simpler and still distinct from copying. The random-partition ablation, together with a measurement of how modular each parent's co-activation graph is at split time, decides this.
Why not just downcycle a dense model once, as ExpertWeaver does? Its one-shot partition needed 200B tokens of recovery training, and it must invent routing from scratch at conversion time. Our splits inherit an already-trained router. The dense-then-split baseline measures whether the coarse MoE stage is worth having over a dense stage of equal compute.
Proposed Experiments and Decisive Checks
We train at two scales: about 1B total parameters with 0.2B active, and about 7B total with 1B active, matching OLMoE-1B-7B's shape. Training uses FineWeb-Edu (The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale) for 50B and 100B tokens respectively. A replication at the small scale on the OLMoE/Dolma mixture checks that results do not depend on the dataset. Evaluation covers validation perplexity per domain, plus ARC-C, ARC-E, HellaSwag, PIQA, SciQ, MMLU, Winogrande, and LAMBADA, which overlap the suites used by SPARKLING and SD-MoE. Routing diagnostics are expert similarity, top-expert token share, and routing churn.
Four checks are decisive. First, at the moment of every split, loss must change by less than measurement noise. This verifies exactness, and it is a check copy-based baselines cannot pass. Second, final perplexity must beat direct 64/top-8 training beyond seed variance at both scales. Third, inter-expert similarity after training should fall below the direct-training level. If perplexity improves while similarity does not, the mechanism we propose is wrong even if the result holds. Fourth, the structured-versus-random partition ablation determines whether the coarse-to-fine borrowing from graph coarsening contributes anything beyond the exact split itself.
As a cheap secondary measurement, we will report wall-clock time for the coarse stages. Top-1 routing over 8 experts dispatches less fragmented traffic than top-8 over 64, which could make early training faster at equal FLOPs.
References
- OLMoE: Open Mixture-of-Experts Language Models
- HARP: Hierarchical Representation Learning for Networks
- A Fast Multi-Scale Method for Drawing Large Graphs
- Learning Topological Representation for Networks via Hierarchical Sampling
- Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
- SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
- ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
- Hierarchical Routing Mixture of Experts
- Hierarchical Mixture-of-Experts with Two-Stage Optimization
- SD-MoE: Spectral Decomposition for Effective Expert Specialization
- Improving Expert Specialization in Mixture of Experts
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale