TL;DR
This paper introduces HookMoE, a lightweight, plug-and-play module that significantly speeds up LLM inference by reducing the number of activated experts in Mixture-of-Experts (MoE) models, while simultaneously compensating for the resulting performance drop. This approach achieves substantial inference acceleration with minimal additional training, making large MoE models more practical for real-world deployment.
What Problem Does This Paper Address?
Large Language Models (LLMs) have become incredibly powerful, but their sheer size leads to high computational costs during inference. Mixture-of-Experts (MoE) architectures offer a solution by activating only a subset of parameters (experts) per token, promising efficiency. However, aggressively reducing the number of activated experts to gain speed often results in a significant degradation of model performance. This paper tackles the critical trade-off between inference efficiency and model accuracy in MoE LLMs, aiming to achieve substantial speedups without sacrificing performance.
Methods
The core contribution is the HookMoE framework, which operates in two stages:
- Layer-Depth Based Dynamic-k Gating (TLNLE): The authors observe that deeper layers in MoE models can often tolerate fewer activated experts without a significant performance hit. They introduce the Top Layer Needs Less Experts (TLNLE) strategy, which systematically reduces the number of active experts in the upper layers of the model. This is achieved by defining a range
[l, r]where the number of active experts is reduced, while the initial layers retain more experts. - Lightweight Hook Compensation Module: To counteract the performance degradation caused by expert reduction, a small, trainable Hook module is inserted before the Feed-Forward layer of a selected transformer block. This module learns to compensate for the lost information by applying a learned transformation. It consists of two linear layers with a ReLU activation and is trained on a small calibration dataset. The Hook module is designed to be "plug-and-play," requiring minimal fine-tuning of only a tiny fraction of the model's parameters.
Datasets
The paper uses a combination of datasets for calibration and evaluation:
- Calibration Datasets:
- C4: A general-domain corpus for broad linguistic coverage.
- MATH: A specialized dataset for mathematical reasoning.
- For calibration, a small subset of 512 examples from each dataset was used, with a sequence length of 2048.
- Evaluation Benchmarks:
- General Tasks: Eight established benchmarks including ARC Challenge, ARC Easy, BoolQ, RTE, HellaSwag, MMLU, OpenBookQA, and Winogrande.
- Domain-Specific Tasks: Two specialized benchmarks: GSM8K (mathematical reasoning) and Minerva Math.
Key Insights and Ideas
- Layer-wise expert sensitivity: Deeper layers in MoE models are less sensitive to the number of activated experts, allowing for targeted expert reduction.
- The "Hook" for compensation: A small, learnable module can effectively "hook" into the MoE architecture to restore performance lost due to expert reduction, acting as a learned correction.
- Minimal fine-tuning is sufficient: The Hook module and TLNLE strategy require only a tiny fraction of the model's parameters to be fine-tuned, making the approach highly efficient.
- Domain-specific calibration matters: Using calibration data that aligns with the target domain (e.g., MATH for math tasks) yields better performance gains than general-domain calibration.
Key Results
- Performance Restoration: HookMoE achieved an average performance degradation of only 2.5% across various benchmarks when reducing activated experts by over 50%.
- Inference Speed-up: The method achieved a 1.42x inference speed-up during the pre-fill stage.
- Parameter Efficiency: Only 0.14% to 0.70% of the total parameters needed fine-tuning.
- Competitive with SOTA: Combined with existing dynamic-k gating methods, HookMoE outperformed them by over 0.6% in overall performance.
- Hardware Acceleration: Speedups were more pronounced on inference-oriented GPUs (H20) compared to compute-oriented GPUs (A800).
Limitations and Future Work
- Stronger compensation with fewer experts: The Hook method shows stronger compensation when the number of activated experts is significantly reduced. As the number of activated experts approaches the original setting, the compensation effect becomes less significant.
- Evaluation on larger models: Due to resource constraints, MoE models with over 141B parameters were not evaluated. The authors plan to conduct experiments on larger models to assess generalizability and scalability.
- Future work: The authors suggest exploring more advanced inference frameworks (like vLLM and SGLang) to further improve speedup gains, as CPU scheduling overhead can be a significant bottleneck. They also aim to investigate the integration of TLNLE with other dynamic-k routing methods.
Summary updated