Expert Routing as a Topic Model: Finding Anchor Experts in Pretrained Mixtures of Experts

Mixture-of-experts language models, which are networks that send each token through only a few of many parallel sub-networks ("experts"), are usually described as divisions of labor in which each expert specializes. Recent audits contradict this. In three open models, a "standing committee" of two to five experts receives most of the routing traffic across every domain, and peripheral experts fire only on narrow terminology (The Illusion of Specialization). Those audits are descriptive. They rank experts by usage, but they cannot say which experts are truly specialized, what they specialize in, or which experts are redundant mixtures of others. This note proposes to answer those questions with an old tool built for a structurally identical problem: geometric topic modeling, which recovers topics as the corners of a simplex enclosing word co-occurrence statistics.

The mapping is literal. Count how often each expert is selected inside each document and you get a documents-by-experts count matrix, which is a bag-of-words corpus whose vocabulary is the set of experts. Anchor-word algorithms (A Practical Algorithm for Topic Modeling with Provable Guarantees) find "anchor words", which are words that appear under only one topic, as the extreme points of normalized co-occurrence rows. Every other word is recovered as a convex mixture of those anchors. Applied to routing, the claim is that a pretrained MoE layer contains a small set of anchor experts that sit at the corners of the routing simplex and mark distinct routing modes. The standing committee should sit in the interior, as near-uniform mixtures of all modes. If so, this decomposition says which experts can be merged or pruned safely, which are interior and nearly reconstructible from others, and which must be kept, namely the anchors. The test is whether this beats usage-frequency and spectral pruning on reasoning benchmarks, where pruning is known to fail.

Core Claim

The quantity we analyze is defined per MoE layer. Let $Q \in \mathbb{R}^{E \times E}$ be the expert co-selection matrix, where $E$ is the number of experts and $Q_{ij}$ is the normalized frequency with which experts $i$ and $j$ are both selected by tokens of the same document. Row-normalize it to $\bar Q$. Under the topic-model reading, there is a matrix $A$ of $K$ routing topics over experts. Here $K$ is the number of latent routing modes, $A_{ik} = p(\text{expert } i \mid \text{mode } k)$, and $R$ is the mode-mode correlation matrix. Together they give

$$Q = A R A^\top ,$$

and every row of $\bar Q$ is a convex combination of the $K$ anchor rows:

$$\bar Q_i = \sum_{k=1}^{K} C_{ik}\, \bar Q_{a_k}, \qquad C_{ik} \ge 0,\ \sum_k C_{ik} = 1,$$

where $a_k$ is the anchor expert of mode $k$ and $C_{ik}$ is expert $i$'s barycentric coordinate, its share in mode $k$.

The proposal makes three falsifiable predictions. First, a small number of experts per layer are near-separable anchors. Second, the standing-committee experts identified by The Illusion of Specialization have high-entropy coordinates $C_i$, meaning they are interior. Third, removing an interior expert and redistributing its traffic by its barycentric coordinates costs less accuracy than removing an equally used expert chosen by frequency or by spectral criteria.

Nothing here requires retraining. That keeps the work cheap and makes its conclusions about existing models.

Relation to Prior Work

The motivation comes from the MoE diagnostics. The standing-committee audit found Gini coefficients above 0.88 and cross-domain Jaccard overlap above 0.85 in OLMoE-1B-7B, Qwen3-30B-A3B, and DeepSeek-V2-Lite, but it offers no generative account of the structure. The "basic-refinement" reading of shared versus routed experts (Decoding Knowledge Attribution in Mixture-of-Experts) suggests the same core-periphery split. We turn that split into a measurable property: it corresponds to an interior-versus-vertex position in the routing simplex.

The method comes from geometric topic modeling. We reuse FastAnchorWords and RecoverKL from A Practical Algorithm for Topic Modeling with Provable Guarantees, which pick anchors by greedily maximizing distance from the span of earlier picks and then recover mixtures by KL projection. We also reuse the separability theory from Learning Topic Models - Going beyond SVD. These methods have two properties we need: they are deterministic and avoid local optima, and they work on an $E \times E$ matrix, so they cost almost nothing. Experts may not be cleanly separable. Shared experts might mean no expert is exclusive to one mode. For that case we fall back on minimum-volume simplex fitting (Minimum Volume Topic Modeling), which needs only the weaker "sufficiently scattered" condition. We also use near-linear-time simplex recovery (Finding a latent k-simplex in O*(k · nnz(data)) time via Subset Smoothing) as a robustness check on token-level rather than document-level counts.

The closest MoE work takes a different route. Eigenvectors of Experts are Training-free Non-collapsing Routers (SSMoE) builds training-free spectral anchors from expert weight matrices and prunes 25% of experts. It reports gains on reasoning and BoolQ but significant degradation on GSM8K. Its anchors are orthogonal axes of the parameters. Ours are extreme points of the routing behavior, so the two make different predictions about which experts are dispensable. The Extra Tokens Matter imposes a minimum-volume simplex on Vision Transformer factor tokens during training. We impose nothing; we test whether such geometry already exists in trained language MoEs. Training-time router redesigns, namely X-MoE's hypersphere routing (On the Representation Collapse of Sparse Mixture of Experts), KERN (Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel), and loss-free balancing (Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts), are complementary. They change how models are trained, and this work measures what trained models contain.

Research Program

We first build routing corpora. Each model is run over a calibration set drawn from FineWeb-Edu (HuggingFaceFW/fineweb) and over MMLU regrouped into the nine domains of the standing-committee audit. For each layer we log top-$k$ selections and form both document-level and token-window-level count matrices.

We then validate the recovery pipeline before trusting it on real data. Following the semi-synthetic protocol in A Practical Algorithm for Topic Modeling with Provable Guarantees, we fit a topic model to real routing counts, resample synthetic routing corpora from it, and confirm that anchors and the matrix $A$ are recovered at the corpus sizes we use.

Next comes the geometric analysis. Per layer, we measure how separable the routing is, using the residual distance of the $K$-th anchor from the span of the previous ones as $K$ grows. A sharp elbow indicates a real number of modes and also addresses the open problem of choosing $K$ that Minimum Volume Topic Modeling flags. We also compute each expert's coordinate entropy, and correlate interior position with the standing-committee membership and contribution index of The Illusion of Specialization. For interpretation, we label routing topics by their dominant MMLU domains and by top tokens.

The last stage turns the geometry into an intervention. To remove expert $i$, we redirect its router logit mass to the anchors in proportion to $C_{ik}$, a barycentric reroute. We compare this against dropping the expert and renormalizing. The pruning order removes experts that are both interior and well reconstructed, meaning low KL residual in RecoverKL, and never removes anchors.

Baselines

The naive reference point is the unpruned model, together with random pruning at the same ratio. These give the ceiling and the floor.

The first real baseline is usage-frequency pruning, which removes the least-selected experts. It is the obvious heuristic, and because of the standing committee it will tend to keep exactly the high-traffic interior experts that our method considers merge-able. That makes it the sharpest contrast.

The strongest existing method is SSMoE spectral routing and pruning (Eigenvectors of Experts are Training-free Non-collapsing Routers). The figure to beat is its reported 8% average reasoning improvement on GPT-OSS-20B with 25% fewer experts, along with its 24% BoolQ gain at 120B. No reported GSM8K number was found on Lacuna for SSMoE beyond the statement that aggressive pruning "significantly degraded performance". We will therefore measure SSMoE's GSM8K accuracy ourselves on OLMoE-1B-7B and DeepSeek-V2-Lite at 25% pruning.

For fairness, every method uses the same calibration tokens, the same pruning ratio, and no fine-tuning afterward. Where SSMoE's mixing weight $\alpha$ needs tuning, it gets the same small validation sweep we use for $K$. Beating SSMoE means matching it within noise on BoolQ and the ARC, HellaSwag, PIQA, SciQ, and WinoGrande suite, while losing significantly less GSM8K accuracy relative to the unpruned model.

Two ablations isolate the key ingredient. One keeps our pruning order but drops experts without barycentric rerouting. The other keeps rerouting but chooses experts by frequency. Together they show whether the gain comes from which experts are chosen or from how their traffic is redistributed.

Why not just…?

Why not just prune by usage frequency? Frequency treats the standing committee as indispensable because it is busy. Our hypothesis is that busyness and irreplaceability are different things: a committee expert may be a near-average of the anchors and therefore reconstructible. The frequency baseline and the ablation that pairs frequency ordering with rerouting test this directly. If frequency ordering performs as well, the geometric account of the committee is still interesting, but its practical value disappears, and we will report that.

Why not just run non-negative matrix factorization, k-means, or Gibbs-sampled LDA on the routing counts? This is a real threat, because anchor methods are a special case of separable matrix factorization. The advantages of anchors are determinism, identifiability under correlated modes (A Practical Algorithm for Topic Modeling with Provable Guarantees), and the fact that they name actual experts as corners rather than abstract factors. Those advantages matter for pruning only if they yield a better ordering. We will run NMF and Gibbs LDA with matched $K$, convert each to an interior score, and compare. If they tie, the contribution narrows to the analysis and its speed.

Why not just use SSMoE's weight eigenvectors? Weight geometry says what an expert can compute, while routing geometry says what it is used for. SSMoE's GSM8K failure suggests weight-based redundancy misses functional roles in reasoning. If our routing-based ordering also fails on GSM8K, that would point to reasoning relying on interior experts. That would be a substantive negative result rather than a tuning issue.

Decisive Checks

The cleanest causal test is a targeted ablation of a single expert. Removing an anchor expert should raise loss mainly on documents with high weight on its routing topic. Removing an interior expert should raise loss diffusely and modestly across all nine MMLU domains, and barycentric rerouting should recover most of that loss. If anchor ablations hurt broadly, or interior ablations are catastrophic, the simplex reading of MoE routing is wrong.

The second check is agreement across models and layers. Separability elbows and the interior placement of committee experts should appear in all three open models. Otherwise the phenomenon reflects one model's training recipe.