TL;DR
This paper introduces GuiLoMo, a novel strategy for optimizing the configuration of LoRA-MoE models by dynamically allocating the number and rank of experts per layer. By employing a bilevel optimization approach with GuidedSelection Vectors, GuiLoMo tailors expert configurations to specific models and tasks, leading to significant performance improvements across various benchmarks.
What Problem Does This Paper Address?
Large Language Models (LLMs) are powerful but can struggle with specific tasks or domains. Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA offer an efficient way to adapt LLMs, but their capacity is limited. Combining LoRA with Mixture-of-Experts (MoE) into LoRA-MoE enhances capacity, but current approaches face two main limitations: 1) they don't consider how downstream tasks influence optimal expert configurations, and 2) they assign a uniform rank to all experts, limiting representational diversity. This uniformity prevents LoRA-MoE from effectively handling diverse and complex inputs. GuiLoMo aims to address these limitations by enabling fine-grained, layer-wise allocation of both expert numbers and ranks.
Methods
GuiLoMo employs a two-step approach:
- Obtaining GuidedSelection Vectors (GSVs): A bilevel optimization process is used to learn GSVs. These vectors guide the selection of optimal expert numbers and ranks, considering both the model backbone and the specific downstream task. This process is designed to be differentiable, allowing for end-to-end optimization.
- Allocating Expert Numbers and Ranks: The learned GSVs are then used to assign the optimal expert numbers and ranks for each module (MHA and FFN) in the LoRA-MoE layers. This is followed by the final fine-tuning of the model.
Key novelties include:
- GuidedSelection Vectors (GSVs): These learned vectors capture task- and model-specific needs for expert configuration.
- Bilevel Optimization: This approach allows the GSVs to be learned in a way that optimizes for the downstream task loss while considering the underlying LLM's parameters.
- Layer-wise, Fine-grained Allocation: Unlike previous methods that used group-wise or uniform allocation, GuiLoMo optimizes expert numbers and ranks for each layer individually.
- Differentiable Expert/Rank Selection: Techniques like Straight-Through Gradient Estimator (STGE) and auxiliary virtual vectors (ME, MR) are used to enable gradient flow through the discrete selection process.
Datasets
The paper evaluates GuiLoMo on a variety of datasets across different task types:
- Natural Language Understanding (NLU):
- GLUE benchmark tasks: MRPC, COLA, RTE
- Question Answering (QA): ScienceQA, CommonsenseQA, OpenBookQA
- Mathematical Reasoning:
- MetaMathQA (for instruction tuning)
- MultiArith, SVAMP, GSM8K
Key Insights and Ideas
- Joint Optimization is Crucial: Optimizing expert numbers and ranks jointly leads to better performance than optimizing either one in isolation.
- Layer-wise Allocation Matters: GuiLoMo's ability to allocate layer-wise optimal expert numbers and ranks significantly improves performance by better exploiting LoRA-MoE's potential.
- MHA vs. FFN Layer Behavior: Multi-head attention (MHA) modules benefit more from increased expert numbers and ranks in the bottom and top layers, while feed-forward networks (FFN) show this trend primarily in the middle and top layers.
- Expert Diversity is Beneficial: Allocating diverse expert ranks enables more flexible and specialized adaptation to different tasks.
- Task Difficulty Influences Allocation: Harder tasks benefit more from increasing the number of experts rather than just raising the rank of existing experts.
Key Results
GuiLoMo consistently outperforms existing state-of-the-art methods across various benchmarks and models (LLaMA-7B, LLaMA-27B, LLaMA-38B, Mistral-v0.17B, LLaMA-213B).
- NLU and QA: GuiLoMo achieves superior or comparable performance to baselines like MOLA and AlphaLoRA. For instance, relative to AlphaLoRA-Uniform(8), GuiLoMo shows consistent gains of 0.61% to 0.84% across different model settings.
- Mathematical Reasoning: GuiLoMo exceeds AlphaLoRA+SoRA by an average of 2.48% to 2.61% on LLaMA-7B and LLaMA-27B.
- Ablation Studies: Removing either adaptive expert allocation or varying rank allocation from GuiLoMo leads to performance drops of 1.50% and 1.10%, respectively, highlighting the importance of both components.
- GPU Memory: GuiLoMo results in only a minimal increase in GPU memory usage compared to standard LoRA-MoE.
Limitations and Future Work
- Model Scale: The current experiments are limited to models up to 13B parameters. Evaluating GuiLoMo on larger open-source LLMs (e.g., LLaMA-70B) is suggested to explore its scalability.
- Task Modality: The evaluation is restricted to standard NLP tasks. Generalizing GuiLoMo to other modalities or multi-modal scenarios is left as future work.
Summary updated