Abstract
Mixture of Experts (MoE) architectures have significantly advanced the state-of-the-art in Large Language Models (LLMs) by enabling high performance with sparse computation. However, their massive parameter counts present substantial deployment and memory overheads. We propose MoE-I², a novel two-stage compression framework that combines Inter-Expert Pruning with Intra-Expert Low-Rank Decomposition. By utilizing genetic algorithms for optimal expert selection and non-uniform rank allocation based on layer-wise sensitivity analysis, MoE-I² achieves over 50% parameter reduction. Our results on Mixtral and DeepSeek models show that the compressed models can maintain or even exceed original performance after lightweight fine-tuning, while delivering up to 1.28x inference speedup.
Introduction
Mixture of Experts (MoE) models like Mixtral and DeepSeek have become foundational for modern LLMs. While computationally efficient during inference due to sparse activation, their total parameter count (e.g., 47B for Mixtral-8x7B) necessitates large memory footprints, complicating deployment on consumer-grade hardware. Existing compression techniques often fail to account for the unique structural redundancies within MoE layers. MoE-I² addresses this by targeting redundancy at both the expert level and within the internal weights of individual experts.
The MoE-I² Framework
The MoE-I² pipeline follows a structured three-stage approach to minimize model size while preserving "expertise."
Stage 1: Inter-Expert Pruning
The first stage involves removing redundant experts entirely. The authors employ a Layer Importance Analysis to determine non-uniform pruning ratios, recognizing that not all layers contribute equally to the model's output. To navigate the massive combinatorial space of expert subsets, a Layer-wise Genetic Search is used to evolve the most effective combinations. Furthermore, a Block-wise KT-Receptive Field evaluation is implemented to ensure pruning decisions are optimized across sequences of layers, preventing local optima.
Stage 2: Intra-Expert Decomposition
Remaining experts undergo further compression via Low-Rank Decomposition. Unlike standard approaches that apply a uniform rank, MoE-I² utilizes non-uniform rank allocation. Importance scores are assigned to experts, allowing "critical" experts to retain higher fidelity while more aggressively compressing those with lower utility. This stage leverages an extension of SVD-LLM to handle the weight matrices within each expert.
Stage 3: Performance Recovery
To mitigate any performance degradation from compression, the framework concludes with a lightweight fine-tuning stage using Low-Rank Adaptation (LoRA). This process allows the model to recalibrate its remaining parameters to the compressed architecture.
Key Insights: Layer Sensitivity and Genetic Search
A pivotal finding of this research is the "Deep Sensitivity" phenomenon. Analysis of normalized loss values across layers reveals that MoE models exhibit a sharp increase in sensitivity as information progresses toward the output layers.
Pruning shallow layers often has negligible impact, whereas pruning deep layers can lead to catastrophic performance loss. This observation justifies the use of non-uniform pruning and decomposition ratios. Additionally, the use of genetic algorithms proved essential; for a model with 64 experts per layer, finding the optimal 16 to keep involves a search space of $C_{64}^{16}$ combinations, which is computationally intractable for brute-force methods.
Experimental Results
The MoE-I² framework was evaluated across nine datasets, including common sense reasoning tasks (e.g., BoolQ, HellaSwag) and language modeling perplexity (WikiText2).
- Compression and Speed: On Mixtral-8x7B, the framework achieved a 51.79% reduction in expert parameters, leading to a 1.28x inference speedup and reducing memory requirements from 87.7 GB to 43.49 GB.
- Performance Gains: Remarkably, after a brief 2-epoch fine-tune, the compressed DeepSeek-V2-Lite model outperformed the original model by 2% on average across zero-shot tasks.
- Efficiency: MoE-I² consistently outperformed baseline pruning methods like Wanda and EEP at equivalent sparsity levels, demonstrating the value of its non-uniform, two-stage approach.
Conclusion
MoE-I² provides a robust solution for the deployment of large-scale MoE models. By combining inter-expert pruning with intra-expert decomposition and respecting the inherent sensitivity of deeper layers, it enables significant hardware efficiency gains without sacrificing model intelligence. Future work will explore scaling this framework to even larger models, such as the 236B parameter DeepSeek-V2.
Reference: Yang, C., Sui, Y., et al. (2024). MoE-I²: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition. arXiv:2411.01016v1.
Summary updated