TL;DR
Upcycling dense neural networks into sparse Mixture-of-Experts (MoEs) is a great way to scale model capacity, but naively copying weights leaves experts homogeneous and the router indecisive. This paper introduces Dirichlet-Prior Shaping Loss (DPSL), a novel regularization technique that forces the router to make confident, highly specialized choices by aligning its routing distributions with a target Dirichlet prior. When applied to upcycled vision-language models, DPSL consistently outperforms standard upcycling and traditional regularizers across multiple multimodal benchmarks.
What Problem Does This Paper Address?
To avoid the massive computational cost of training large models from scratch, researchers often turn to sparse upcycling. This technique takes a pre-trained dense model and initializes a sparse Mixture-of-Experts (MoE) architecture by replicating the dense model's feedforward network (FFN) weights across multiple "experts."
However, because all the experts start out identical, the router module faces a severe cold-start problem. It struggles to differentiate between them, resulting in low-confidence, weakly differentiated routing where the assignment probabilities for every expert hover lazily around $1/N$ (where $N$ is the total number of experts). Standard MoE regularizers like load-balancing loss or z-loss keep the training stable and prevent expert collapse, but they don't actually force these identically initialized experts to specialize.
Methods
The authors propose Dirichlet-Prior Shaping Loss (DPSL) to directly control the statistical properties of the router's outputs.
Instead of just tracking token counts or penalizing large logits, DPSL shapes the entire categorical probability distribution of the router. It leverages a key mathematical property of the Dirichlet distribution: its marginal distributions follow a Beta distribution.
During training, DPSL computes the empirical cumulative distribution function (CDF) of the router's assignments across a batch of tokens. It then uses the Cramér-von Mises criterion to minimize the squared distance between this empirical CDF and the theoretical CDF of a target Beta distribution derived from a chosen Dirichlet prior.
DPSL provides two main flavors of control:
- Symmetric Priors: By setting uniform concentration parameters ($\alpha$), the router is encouraged to make highly confident but globally balanced expert selections across the full probability range.
- Asymmetric Priors: By skewing the concentration parameters based on the input type, developers can inject inductive biases. For example, in a Vision-Language Model (VLM), you can gently guide specific experts to prefer vision tokens and others to prefer language tokens without hard-coding rigid routing rules.
Datasets
The authors evaluated their approach by upcycling vision-language models across three distinct training stages:
- Stage I (Projector Alignment): LLaVA 1.5-558k
- Stage II (MoE Warm-up): A massive 1,206k-token corpus consisting of LLaVA 1.5-mix-665k (30%), SAM (30%), Wikiart (30%), LVIS, ALLaVA, and TextVQA.
- Stage III (Task Finetuning): LLaVA 1.5-mix-665k, SAM, and Wikiart (750k tokens total).
Downstream performance was rigorously tested across six standard vision-language benchmarks: GQA, TextVQA, VizWiz (Test-Dev2024), MME (Perception & Cognition), MM-Vet, and MMBench.
Key Insights and Ideas
- Traditional Regularizers Compress the Router: Visualizing the routing simplex reveals that methods like z-loss actually compact the routing distribution more toward the center, worsening the low-confidence issue. DPSL successfully pushes assignments toward the edges, utilizing the full dynamic range.
- Soft Modality Priors Beat Rigid Rules: Manually locking certain experts to vision tokens and others to text tokens during warm-up hurts performance. It prevents the model from sharing cross-modal knowledge early on. Using asymmetric Dirichlet priors acts as a "soft guide," allowing the model to naturally discover optimal, flexible specialization patterns.
- Intrinsic Load Balancing: Even though DPSL does not contain an explicit load-balancing mathematical term, matching a symmetric Dirichlet prior naturally yields a low Coefficient of Variation (CoV) across experts, ensuring balanced utilization across all layers.
Key Results
The authors integrated DPSL into upcycled MoE variants of state-of-the-art LLM backbones (Qwen2-1.5B, Phi3-mini 3.8B, and Llama3.2-1B) using both standard (2in4) and granular (8in16) MoE configurations.
- Consistent Performance Gains: DPSL consistently achieved the highest average downstream accuracy across all backbones and configurations. For instance, on the Qwen2-1.5B (2in4) setup, DPSL reached an average score of 38.17, comfortably beating naive sparse upcycling (36.17) and the dense baseline (36.60).
- Confident Routing Distributions: As shown in the routing score distributions, standard methods peak sharply at a uniform probability of 0.25. DPSL spreads the scores widely, proving that the router is making distinct, confident choices for individual tokens.
Limitations and Future Work
- Computational Overhead: Calculating the empirical CDF across thousands of tokens introduces a 10–15% computational overhead compared to standard loss functions. To mitigate this, the authors only applied DPSL during the warm-up phase and disabled it during final fine-tuning.
- Evaluation Constraints: Due to the extreme financial and environmental costs of training multiple MoE variants, the authors were unable to run evaluations across multiple random seeds for the entire matrix of backbones.
- Broader Applications: The authors note that DPSL is a general tool. Future work includes applying it to training MoEs completely from scratch, or using it in entirely different neural network modules that output categorical probability distributions.
Summary updated