Examining the Experts: Measuring Specialization and Collapse in Mixture-of-Experts Models with Item Response Theory

A mixture-of-experts (MoE) language model splits each feedforward layer into many sub-networks, the "experts," and uses a small learned router to send each token to a few of them. Training such models suffers from expert collapse, which takes two forms that pull against each other. In starvation, a few experts take most tokens while the rest barely train. In homogenization, balancing penalties spread tokens so evenly that every expert ends up learning the same thing. The metrics now used to track these failures cannot tell the two cases apart. Load imbalance, routing entropy, and representation overlap all fail to separate an expert that is generally weak from an expert that is strong on a narrow set of tokens and weak elsewhere. Psychometrics solved the same confusion for human tests decades ago. Item response theory (IRT), the statistical model behind standardized exams, separates a test-taker's ability from an item's difficulty, and it separates general ability from specific skills.

We propose to give MoE experts a real exam. On a small set of probe tokens, we force each token through each expert in turn and record whether that expert's output lowers next-token loss. We then fit a multidimensional IRT model to the resulting expert-by-token response matrix. Once general expert quality and token difficulty are removed, the share of structure left over is a direct measure of functional specialization. Two parts of IRT make this cheap enough to run during training. Item discrimination tells us which tokens are informative about experts, and adaptive testing uses this to choose a few hundred probe tokens instead of thousands. The central test is whether this index, measured early in training, predicts which balancing method and strength will give the best final model better than cheaper signals do, including plain early validation loss.

Core Claim and What Is New

The obvious way to combine these directions would be to build the router itself as an IRT model, scoring token $t$ for expert $e$ as expert ability minus token difficulty. We argue this is a dead end, and the argument shapes the proposal. In a multidimensional IRT logit $a_t^\top\theta_e - b_t$, the difficulty $b_t$ is shared by all experts, so it cancels inside the router's softmax and top-$k$ choice. What remains is a bilinear score, which is exactly what a linear router already computes. The IRT structure only matters when there is an observed response to model. The router has no such response. The experts' actual effect on loss does.

The closest existing work is IRT-Router. It treats whole LLMs as test-takers and user queries as items, and uses IRT to decide which black-box model answers each prompt. It works at the system level, never touches expert parameters, and has no notion of collapse. Our work differs in three ways. The test-takers are internal experts. The items are tokens at a given layer. The purpose is measurement and diagnosis of training dynamics, not choosing where to send a query.

Concretely, for layer $\ell$, let $Y_{e,t}=1$ if replacing the layer's MoE output with expert $e$ alone gives lower next-token loss on token $t$ than a uniform mix of all experts, and $Y_{e,t}=0$ otherwise. We fit

$$ P(Y_{e,t}=1) = \sigma\big(a_t^\top \theta_e - b_t\big), $$

where $\theta_e \in \mathbb{R}^d$ is expert $e$'s ability vector, $a_t$ is token $t$'s discrimination vector (how sharply it separates experts along each dimension), and $b_t$ is its difficulty. The first dimension is anchored as general ability. We define the specialization index $S_\ell$ as the fraction of deviance explained by dimensions $2,\dots,d$ beyond the one-dimensional (Rasch-style) model. This model has only general ability and token difficulty. The diagnosis follows from where the structure sits:

  • Homogenization: $S_\ell$ is near zero, because the one-dimensional model already fits and experts differ only in overall quality.
  • Starvation: the spread of the general-ability coordinate is large while $S_\ell$ stays small, because some experts are simply worse.
  • Healthy specialization: $S_\ell$ is high while general ability stays tight across experts.

A second quantity is router faithfulness, the correlation between router scores and the fitted interaction term $a_t^\top\theta_e$. It directly tests the claim in Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss that routers are blind to what experts can do.

Motivation and Constraints from Prior Work

Three MoE papers motivate the gap by showing that specialization is the quantity that matters but is poorly measured.

  • Advancing Expert Specialization for Better MoE names the paradox that load-balancing losses cause homogenization. It fixes this with orthogonality and variance losses and reports that expert overlap fell by up to 45%. That overlap is measured geometrically on expert outputs, and orthogonal outputs need not be useful outputs.
  • Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss shows there is a "Goldilocks zone" of specialization strength that is found only by empirical tuning. An early-training index that locates this zone is exactly what our proposal tests.
  • Expert Divergence Learning enforces specialization by pushing apart the routing distributions of labeled domains. It needs those labels, and it degraded when documents were treated as domains. Our index needs no labels, since its dimensions are learned from responses.

Three balancing methods are the training regimes we will measure. Auxiliary-Loss-Free Load Balancing adjusts per-expert routing biases from observed load, with no gradient penalty. $\phi$-Balancing balances load at the population level using a running average of expert load. BASE Layers solves token assignment as a linear assignment problem. All three control load, so all three can reduce starvation. None of them measures whether the balanced experts learned different things, which is the question our index answers.

The architectural work in DeepSeekMoE adds always-on shared experts meant to absorb common knowledge. This predicts that shared experts should load almost entirely on the general-ability dimension, and routed experts on the interaction dimensions. That gives a checkable prediction on a public model.

The method comes from psychometric-neural work:

  • DIRT predicts item difficulty and discrimination from item text with a neural encoder. We reuse this to predict $a_t, b_t$ from the token's hidden state, which lets the model score tokens it has never probed.
  • QIKT shows that an IRT output layer with separate components stays interpretable without losing accuracy.
  • Disentangled Knowledge Tracing shows that unless ability and difficulty are separated explicitly, models confound them under biased data. Here the "biased data" is the router's own non-random token assignment.

The last point is also a real constraint. Response data drawn only from tokens the router already sends to an expert would be confounded, so we force-route probe tokens to every expert rather than observing natural traffic.

Research Program

Build the probe. The first component is a probe harness for a trained or training MoE. At a chosen layer, it replaces the MoE output with each single expert's output, rescaled to match the router-weighted norm, runs the rest of the network, and records the binary response.

Fit the IRT model. We fit the multidimensional model above by marginal maximum likelihood with a hidden-state encoder for $a_t, b_t$, in the style of DIRT. We choose $d$ by held-out response likelihood.

Make it cheap with adaptive testing. The cost of forced routing grows with the number of experts times the number of probe tokens. To reduce it, we borrow computerized adaptive testing. At each checkpoint we take the current ability estimates and choose the probe tokens with the highest Fisher information about $\theta$. We then measure how many probe tokens are needed to estimate $\theta_e$ to a fixed standard error, compared with random probe sampling. If adaptive probing cuts this by a large factor, the index can be tracked every few thousand steps at negligible cost.

Train the models. We will train a grid of small MoE language models, about 0.5–1B total parameters with 16 and 64 experts, top-2 routing, and about 20B tokens, on FineWeb. Each run uses the same architecture and data order. The grid varies the balancing method and its strength, giving roughly thirty runs.

Baselines

Naive reference point. The first reference is the model's own early validation loss. The practical question is "which of these runs will end best?", and extrapolating from early loss is what practitioners already do. The index earns its keep only if, at 10% of training, it rank-orders final downstream quality better than loss at 10%.

Existing collapse metrics. Next come the metrics in current use, all computed on the same checkpoints and the same probe tokens:

These are the strongest competitors because they are free to compute and are what papers report today. No reported result was found on Lacuna for any of them as an early predictor of final downstream performance across balancing regimes, so the comparison sets its own reference.

Ablations. Two ablations isolate what IRT adds. The first applies double-centering and SVD to the same forced-routing loss matrix. This is a linear, link-free decomposition that uses the identical data, so it separates "forced-routing responses help" from "the IRT model helps." The second uses random rather than adaptive probe selection at matched probe counts.

What counts as beating them. Every predictor gets the same checkpoints, the same probe budget, and a leave-one-run-out regression to final scores. The IRT index must show higher Spearman correlation with final scores than every baseline, with a bootstrap interval over runs that excludes the best baseline's value.

Why not just…?

Why not just use an IRT-parameterized router? Token difficulty is shared across all experts, so it cancels in the router's softmax. The resulting router is a reparameterized linear router. The IRT structure pays off only where there is an observed outcome to model, which is why we apply it to measured expert effects rather than to routing decisions.

Why not just add the orthogonality/variance losses or the ERC loss and skip measurement? These are interventions, and both papers report that their strength must be tuned: ERC warns explicitly that over-specialization hurts. Our index is not a competitor to these losses. It is a candidate tool for setting their weight. The experiment that settles this uses our index at 10% of training to pick each method's coefficient, then checks whether that choice matches the grid's best final coefficient more often than a choice made by early loss.

Why not just take an SVD of the forced-routing loss matrix? This is the real threat. If double-centered SVD predicts as well as the IRT index, then the value lies in the forced-routing probe and not in psychometrics. We would report that result as the finding. IRT has two specific advantages that the ablation tests directly. The logistic link keeps easy and hard tokens from dominating the decomposition. The information function enables adaptive probe selection, which SVD cannot provide. If adaptive probing does not reduce cost and the link adds no predictive power, the psychometric framing should be dropped.

Why not just use domain labels, as Expert Divergence Learning does? Labels exist only at a coarse level. EDL reports that finer, document-level groupings hurt. Our index finds its dimensions from responses. We will check whether they align with FineWeb topic clusters that are held out and never used in fitting.

Proposed Experiments and Decisive Checks

Planted-structure test. Before any correlational claim, we train a 16-expert model in which each expert group is forced to see only one of four synthetic domains, and compare it to a model with random token assignment. The IRT model should recover four interaction dimensions in the first model and none in the second. Separately, we apply stronger auxiliary balancing to show the index falls toward zero as homogenization sets in. If either check fails, the index is not measuring what we claim.

Main test. On the thirty-run grid, we compare early-checkpoint predictors against final validation perplexity and against HellaSwag, ARC-Challenge, PIQA, and MMLU accuracy. We expect, but do not assume, that:

  • no balancing gives low index values from starvation;
  • strong auxiliary loss gives low index values from homogenization;
  • loss-free and $\phi$-balancing sit higher.

For context, $\phi$-Balancing reports Math500 rising from 64.8% to 66.2% on Moonlight-16B-A3B. That gain is at a scale beyond our grid, and we cite it only as evidence that balancing choice changes downstream reasoning.

Router-faithfulness test. We measure how well router scores track the interaction term across regimes, and whether ERC-trained models score higher, as their paper predicts.

External check. We probe public DeepSeekMoE-16B checkpoints to test whether shared experts load on the general-ability dimension and routed experts on specialization dimensions, as DeepSeekMoE's design intends.

Efficiency. We report probe cost as a fraction of training compute, and the number of adaptive versus random probe tokens needed to reach a fixed standard error on $\theta$.