Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs

Leyla Mirvakhabova, Babak Ehteshami Bejnordi, Gaurav Kumar, Hanxue Liang, Wanru Zhao, Paul Whatmough

TL;DR

Upcycling dense neural networks into sparse Mixture-of-Experts (MoEs) is a great way to scale model capacity, but naively copying weights leaves experts homogeneous and the router indecisive. This paper introduces Dirichlet-Prior Shaping Loss (DPSL), a novel regularization technique that forces the router to make confident, highly specialized choices by aligning its routing distributions with a target Dirichlet prior. When applied to upcycled vision-language models, DPSL consistently outperforms standard upcycling and traditional regularizers across multiple multimodal benchmarks.

What Problem Does This Paper Address?

To avoid the massive computational cost of training large models from scratch, researchers often turn to sparse upcycling. This technique takes a pre-trained dense model and initializes a sparse Mixture-of-Experts (MoE) architecture by replicating the dense model's feedforward network (FFN) weights across multiple "experts."

However, because all the experts start out identical, the router module faces a severe cold-start problem. It struggles to differentiate between them, resulting in low-confidence, weakly differentiated routing where the assignment probabilities for every expert hover lazily around $1/N$ (where $N$ is the total number of experts). Standard MoE regularizers like load-balancing loss or z-loss keep the training stable and prevent expert collapse, but they don't actually force these identically initialized experts to specialize.

Methods

The authors propose Dirichlet-Prior Shaping Loss (DPSL) to directly control the statistical properties of the router's outputs.

Instead of just tracking token counts or penalizing large logits, DPSL shapes the entire categorical probability distribution of the router. It leverages a key mathematical property of the Dirichlet distribution: its marginal distributions follow a Beta distribution.

During training, DPSL computes the empirical cumulative distribution function (CDF) of the router's assignments across a batch of tokens. It then uses the Cramér-von Mises criterion to minimize the squared distance between this empirical CDF and the theoretical CDF of a target Beta distribution derived from a chosen Dirichlet prior.

Figure 1: Sparse upcycling (left) initializes identical experts, yielding homogeneous routing probabil- ities and limited specialization. (right) Our proposed Dirichlet-Prior Shaping Loss guides routing towards desired distributions fostering balanced and confident selection (via symmetric priors) or targeted, modality-/task-aware specialization (via asymmetric priors).

DPSL provides two main flavors of control:

  1. Symmetric Priors: By setting uniform concentration parameters ($\alpha$), the router is encouraged to make highly confident but globally balanced expert selections across the full probability range.
  2. Asymmetric Priors: By skewing the concentration parameters based on the input type, developers can inject inductive biases. For example, in a Vision-Language Model (VLM), you can gently guide specific experts to prefer vision tokens and others to prefer language tokens without hard-coding rigid routing rules.

Datasets

The authors evaluated their approach by upcycling vision-language models across three distinct training stages:

  • Stage I (Projector Alignment): LLaVA 1.5-558k
  • Stage II (MoE Warm-up): A massive 1,206k-token corpus consisting of LLaVA 1.5-mix-665k (30%), SAM (30%), Wikiart (30%), LVIS, ALLaVA, and TextVQA.
  • Stage III (Task Finetuning): LLaVA 1.5-mix-665k, SAM, and Wikiart (750k tokens total).

Downstream performance was rigorously tested across six standard vision-language benchmarks: GQA, TextVQA, VizWiz (Test-Dev2024), MME (Perception & Cognition), MM-Vet, and MMBench.

Key Insights and Ideas

  • Traditional Regularizers Compress the Router: Visualizing the routing simplex reveals that methods like z-loss actually compact the routing distribution more toward the center, worsening the low-confidence issue. DPSL successfully pushes assignments toward the edges, utilizing the full dynamic range.
  • Soft Modality Priors Beat Rigid Rules: Manually locking certain experts to vision tokens and others to text tokens during warm-up hurts performance. It prevents the model from sharing cross-modal knowledge early on. Using asymmetric Dirichlet priors acts as a "soft guide," allowing the model to naturally discover optimal, flexible specialization patterns.
  • Intrinsic Load Balancing: Even though DPSL does not contain an explicit load-balancing mathematical term, matching a symmetric Dirichlet prior naturally yields a low Coefficient of Variation (CoV) across experts, ensuring balanced utilization across all layers.

Key Results

The authors integrated DPSL into upcycled MoE variants of state-of-the-art LLM backbones (Qwen2-1.5B, Phi3-mini 3.8B, and Llama3.2-1B) using both standard (2in4) and granular (8in16) MoE configurations.

  • Consistent Performance Gains: DPSL consistently achieved the highest average downstream accuracy across all backbones and configurations. For instance, on the Qwen2-1.5B (2in4) setup, DPSL reached an average score of 38.17, comfortably beating naive sparse upcycling (36.17) and the dense baseline (36.60).
  • Confident Routing Distributions: As shown in the routing score distributions, standard methods peak sharply at a uniform probability of 0.25. DPSL spreads the scores widely, proving that the router is making distinct, confident choices for individual tokens.
Routing score distributions across different methods

Limitations and Future Work

  • Computational Overhead: Calculating the empirical CDF across thousands of tokens introduces a 10–15% computational overhead compared to standard loss functions. To mitigate this, the authors only applied DPSL during the warm-up phase and disabled it during final fine-tuning.
  • Evaluation Constraints: Due to the extreme financial and environmental costs of training multiple MoE variants, the authors were unable to run evaluations across multiple random seeds for the entire matrix of backbones.
  • Broader Applications: The authors note that DPSL is a general tool. Future work includes applying it to training MoEs completely from scratch, or using it in entirely different neural network modules that output categorical probability distributions.

Summary updated

Abstract

Upcycling pre-trained dense models into sparse Mixture-of-Experts (MoEs) efficiently increases model capacity but often suffers from poor expert specialization due to naive weight replication. Our analysis reveals that upcycled MoEs, even with conventional regularization, exhibit low-confidence, weakly differentiated routing, hindering performance. We introduce Dirichlet-Prior Shaping Loss (DPSL), a novel router regularization technique that directly shapes routing probability distributions by matching expert assignments to a target Dirichlet prior. DPSL offers fine-grained control over expert balance and specialization, and enables encoding of inductive biases such as encouraging experts to focus on specific modalities or tasks, without requiring manual intervention; notably, DPSL is a general tool applicable to any module that outputs categorical probability distributions, extending its utility beyond MoE training. Experiments on upcycled MoE vision-language models (with Qwen2, Phi3, Llama3.2 LLM backbones) show DPSL consistently outperforms upcycling strategies and regularization techniques across standard vision-language benchmarks, addressing the critical issue of poor specialization and fostering more adaptive, higher-performing models.
BibTeX
@inproceedings{mirvakhabova2026dirichlet,
  author = {Leyla Mirvakhabova and Babak Ehteshami Bejnordi and Gaurav Kumar and Hanxue Liang and Wanru Zhao and Paul Whatmough},
  title = {{Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=BfkrQiC0Ss},
}
Figure