Abstract
As Large Language Models (LLMs) continue to scale, the computational and environmental costs associated with their deployment have become increasingly prohibitive. Traditional approaches to efficiency, such as pruning and fixed-rate Mixture-of-Experts (MoE), often suffer from permanent knowledge loss or rigid computational overhead. This paper introduces Dynamic Sparse Mixture-of-Experts (DSMoE), a framework that transforms pre-trained dense LLMs into efficient sparse models. By partitioning existing Feed-Forward Network (FFN) layers into experts and employing a sigmoid-based dynamic gating mechanism, DSMoE enables input-adaptive computation that preserves model intelligence while significantly reducing FLOPs.
Introduction
The pursuit of larger models has led to remarkable capabilities in natural language processing, but at the cost of extreme resource consumption. To mitigate this, researchers typically turn to two strategies: pruning, which removes weights permanently, and Mixture-of-Experts (MoE), which activates a subset of parameters during inference. However, pruning is often "destructive," leading to irreversible loss of pre-trained knowledge. Meanwhile, standard MoE models typically utilize "Top-K" routing, where every token—regardless of its complexity—activates a fixed number of experts. This lack of granularity means simple tokens (e.g., "the") receive the same computational budget as complex technical terms.
DSMoE addresses these limitations by introducing a method to reorganize dense layers into experts and implementing a routing mechanism that adaptively adjusts computation based on the difficulty of the input token.
Methodology
The DSMoE framework is built upon three primary innovations designed to maximize efficiency without sacrificing the performance of the original dense model.
1. FFN Partitioning
Unlike traditional MoE models that require training experts from scratch, DSMoE leverages the existing knowledge within a pre-trained dense model (such as LLaMA). The Feed-Forward Network (FFN) layers are sliced along their intermediate dimensions, reorganizing the original dense weight matrices into $n$ parallel experts. This ensures that the total parameter count remains identical to the original model, but the architecture is now structured for sparse activation.
2. Dynamic Routing with Sigmoid Activation
Standard MoE models use a Softmax-based router to select the "Top-K" experts. DSMoE replaces this with a Sigmoid-based gating mechanism. In this setup, an expert is activated only if its gating value exceeds a learnable threshold $\tau$. This allows for true input-adaptive computation: for a single token, the model may activate zero, one, or all available experts depending on the necessity of the information stored within them.
3. Optimization via STE and Sparsity Loss
To prevent the "dead expert" problem—where inactive experts fail to receive gradients and thus never improve—the authors utilize a Straight-Through Estimator (STE). This allows gradients to flow through the non-differentiable thresholding function during training. Additionally, an $L_1$ sparsity loss is added to the training objective, acting as a regularizer that encourages the model to minimize expert activation while maintaining accuracy.
Experimental Setup
The DSMoE models were evaluated by continually pre-training on 10 billion tokens across diverse, high-quality domains:
- General: Fineweb-edu
- Mathematics: OpenWebMath
- Code: StarCoder
- Synthetic Data: Cosmopedia
Performance was measured across a suite of benchmarks, including HellaSwag, PIQA, and Winogrande for reasoning, as well as TriviaQA and GSM8K for generation and problem-solving.
Key Insights and Results
The "W-shaped" Activation Pattern
An analysis of the model's behavior revealed a unique "W-shaped" activation pattern across layers. DSMoE naturally learns to allocate more experts at the input layers (to capture initial features), the middle layers (to handle complex dependencies), and the output layers (to ensure generation quality). The intervening layers utilize fewer experts, optimizing the computational flow.
Performance Metrics
DSMoE demonstrated superior efficiency and accuracy compared to existing baselines:
- Perplexity: On LLaMA-7B, DSMoE achieved a perplexity of 3.91, significantly better than LLaMA-MoE (4.12) and various pruning methods (~4.01) under similar computational constraints.
- Downstream Tasks: DSMoE outperformed competitors in 7 out of 10 benchmarks at both the 1B and 7B scales. It showed particular strength in complex generation tasks like TriviaQA and GSM8K, where it surpassed SparseGPT and LLM-Pruner.
- Ablation Studies: The importance of the STE was confirmed; without it, the gating mechanism failed to converge, resulting in a perplexity jump from 7.41 to 12.75.
Conclusion and Future Work
DSMoE provides a robust pathway for converting dense LLMs into sparse, adaptive models that preserve pre-trained knowledge. While the current study focused on models up to 7B parameters, the authors anticipate even greater benefits at larger scales (e.g., 70B+), where parameter redundancy is more prevalent. Future research will focus on developing specialized hardware kernels to translate theoretical FLOP reductions into real-world wall-clock speedups and exploring the use of the activation threshold $\tau$ as a dynamic "knob" for real-time speed-accuracy trade-offs during inference.
Summary updated