TL;DR
MergeMoE rethinks how to compress Mixture-of-Experts (MoE) models by framing expert merging as an output-approximation problem rather than simple parameter averaging. By decoupling the SwiGLU expert computation and solving for compression matrices using closed-form linear least squares, MergeMoE consistently outperforms prior expert-merging baselines across multiple open-source MoE models without retraining.
What Problem Does This Paper Address?
Mixture-of-Experts (MoE) architectures achieve high parameter efficiency by routing each input token to a sparse subset of feed-forward "experts." While this drastically reduces computational FLOPs during inference, it requires storing billions of parameters in VRAM, making deployment difficult on memory-constrained hardware.
Prior MoE compression methods like M-SMoE group experts and merge them by heuristically computing weighted averages of their weight matrices. However, parameter-space averaging lacks a firm theoretical foundation and fails to account for the non-linear transformations inside expert layers, leading to significant performance degradation.
Methods
MergeMoE shifts the perspective from merging expert parameters to merging expert outputs:
- Theoretical Formulation: The forward pass of an MoE layer is rewritten using routing and clustering assignment matrices ($A$ and $B$). A merged expert $E'_i$ is designed to approximate a linear combination of original expert outputs $\sum_j B_{ji} E_j(X)$.
- Dimension-Reduction Decomposition: For SwiGLU experts ($E(X) = W_D(\sigma(W_G X) \odot W_U X)$), preserving output equality without reducing dimensions would blow up the parameter size. MergeMoE introduces three dimension-reduction matrices ($T_1, T_2, T_3$) to map concatenated expert matrices back to standard expert dimensions.
- Decoupled Optimization:
- Gating and Up-projections ($T_2, T_3$): Because non-linear activations and Hadamard products make joint optimization intractable, $T_2$ and $T_3$ perform weighted averaging based on expert usage frequency. The authors mathematically prove (Theorem 1) that weighting by relative frequency is optimal for minimizing output error.
- Down-projection ($T_1$): Given a small set of calibration inputs $\hat{X}$, the authors formulate the recovery of $T_1$ as a linear least-squares problem, deriving a closed-form solution via the Moore-Penrose pseudoinverse ($T_1 = QP^\dagger$).
- Execution Pipeline: Experts are clustered by the cosine similarity of their concatenated $[W_U, W_G]$ matrices around the most frequently activated experts. Merging is performed layer-by-layer backwards (from output to input) on a single GPU using intermediate activation hooks.
Datasets
The authors evaluate MergeMoE on seven standard NLP benchmarks:
- WinoGrande (coreference resolution)
- ARC Easy & ARC Challenge (scientific reasoning)
- HellaSwag (commonsense reasoning)
- PIQA (physical interaction QA)
- SQuAD (extractive question answering)
- MRPC (paraphrase identification)
- IFEval (instruction following, paired with ShareGPT for optional distillation)
Note: No new datasets were introduced.
Key Insights and Ideas
- Outputs matter more than weights: Directly averaging weights ignores the non-linear interaction between SwiGLU gating and up-projection. Viewing merging as function approximation in the output space enables rigorous mathematical formulation.
- Frequency weighting is provably optimal: The paper provides formal theoretical backing showing that under output-error bounds, weighting merged experts by activation frequency minimizes the Quasi-Frobenius error bound.
- Closed-form calibration rescues performance: Optimizing the down-projection matrix $W_D T_1$ with a handful of forward-pass activations acts as an error corrector for the non-linear activations in $W_G$ and $W_U$.
- Layer distribution vs. depth: Compressing fewer experts across more layers preserves downstream reasoning capability significantly better than heavily collapsing experts within a few layers.
Key Results
Experiments evaluated DeepSeekMoE (16B $\rightarrow$ 12B), Qwen1.5-MoE-A2.7B (14B $\rightarrow$ 10B), and Qwen3-30B-A3B (30B $\rightarrow$ 25B) against baselines like Average, ZipIt, and M-SMoE:
- Qwen1.5-MoE: MergeMoE outperforms the previous state-of-the-art (M-SMoE) by +1.50 on WinoGrande (70.48 vs. 68.98), +2.71 on PIQA (79.27 vs. 76.56 / ZipIt 77.80), and +2.45 on MRPC (74.75 vs. 72.30).
- Qwen3-30B: MergeMoE achieves 83.04 on ARC-e, 63.48 on ARC-c, and 74.93 on HellaSwag—narrowing the gap to the uncompressed model (84.89, 67.49, 76.38) and comfortably beating the Qwen3-4B dense baseline.
- Low Overhead: Merging layers on Qwen1.5 takes less than 60 seconds on a single NVIDIA H20 GPU.
Limitations and Future Work
- Sample Size Sensitivity: MergeMoE requires calibration data. Performance collapses to random guessing if fewer than ~32–36 samples are used, though accuracy stabilizes and scales smoothly beyond 40–64 samples.
- Cross-Task Transfer Gap: Using calibration samples from a single dataset yields minor performance degradation when evaluating on out-of-domain tasks compared to task-specific calibration.
- Rare Expert Compression: Because the weighting scheme favors frequently activated experts, rarely used experts in the long-tail distribution risk being disproportionately eroded.
Summary updated