GraphLoRA: Empowering LLMs Fine-Tuning via Graph Collaboration of MoE

Ting Bai, Yue Yu, Le Huang, Zenan Xu, Zhe Zhao, Chuan Shi

Abstract

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method that has been widely adopted in various downstream applications of LLMs. Together with the Mixture-of-Expert (MoE) technique, fine-tuning approaches have shown remarkable improvements in model capability. However, the coordination of multiple experts in existing studies solely relies on the weights assigned by the simple router function. Lack of communication and collaboration among experts exacerbate the instability of LLMs due to the imbalance load problem of MoE. To address this issue, we propose a novel MoE graph-based LLM fine-tuning framework GraphLoRA, in which a graph router function is designed to capture the collaboration signals among experts by graph neural networks (GNNs). GraphLoRA enables all experts to understand input knowledge and share information from neighbor experts by aggregating operations. Besides, to enhance each expert's capability and their collaborations, we design two novel coordination strategies: the Poisson distribution-based distinction strategy and the Normal distribution-based load balance strategy. Extensive experiments on four real-world datasets demonstrate the effectiveness of our GraphLoRA in parameter-efficient fine-tuning of LLMs, showing the benefits of facilitating collaborations of multiple experts in the graph router of GraphLoRA.

Introduction

The Mixture-of-Experts (MoE) architecture is the engine behind many state-of-the-art Large Language Models (LLMs). It works by routing input tokens to a small subset of specialized sub-networks, known as experts. However, standard MoE models often suffer from a chronic "load imbalance" issue. Because the router—the gatekeeper that decides which expert gets which token—is usually a simple linear function, it often falls into a "rich-get-richer" trap. A few experts become over-trained and over-utilized, while others remain under-utilized or "lazy."

While researchers have previously attempted to fix this with mathematical penalties (load-balancing losses), these are often blunt instruments. They force experts to take work but do not facilitate effective communication or specialization. The result is often an unstable training process and inefficient learning. GraphLoRA addresses these challenges by reimagining the MoE router as a Graph Neural Network (GNN), allowing experts to collaborate rather than compete in isolation.

Methods: The Graph Router and Coordination Strategies

GraphLoRA replaces the standard linear router with a sophisticated Graph Router. This architecture treats input tokens and experts as nodes in a graph. The input token node is connected to all expert nodes, and expert nodes are connected to each other, with the density of these connections controlled by a hyperparameter.

The GraphLoRA architecture (GMoE) compared to standard LoRA and MoE-LoRA, highlighting the Graph Router and distribution-based coordination strategies.

Instead of a simple dot product, GraphLoRA uses a GNN to aggregate information. This allows the router to capture "collaboration signals," essentially letting the system decide which experts work best together for a specific type of input. To further enhance this collaboration, the framework introduces a two-pronged coordination strategy:

  1. Poisson Distribution-based Distinction Strategy: This strategy uses a loss function to encourage the router's output to follow a Poisson distribution. This helps experts specialize in distinct aspects of the input data, enhancing the model's "distinguish capability."
  2. Normal Distribution-based Balance Strategy: To ensure no expert is left behind, this strategy regulates activation frequencies so they follow a natural normal distribution. This prevents the "lazy expert" syndrome and ensures a balanced load across the network.

By implementing these experts as lightweight LoRA (Low-Rank Adaptation) adapters, GraphLoRA maintains high parameter efficiency while significantly improving model capability.

Experimental Evaluation

The researchers evaluated GraphLoRA across four diverse benchmark datasets to test different reasoning capabilities: ARC-Challenge (complex science questions), OpenBookQA (elementary science facts), SIQA (social commonsense), and BoolQ (reading comprehension). These tests were performed on three popular base models: Llama3-8B, Qwen2-7B, and Yi-1.5-9B.

Key Insights and Results

GraphLoRA consistently outperformed state-of-the-art baselines like MixLoRA and LoRAMoE across almost all datasets. For example, on Qwen2-7B, it reached an average accuracy of 83.29%. Beyond raw accuracy, the framework demonstrated remarkable training stability. Measured by standard deviation across multiple runs, GraphLoRA was significantly more reliable than other sparse MoE methods.

Sensitivity analysis of GraphLoRA (GMoE) across ARC-Challenge and BoolQ datasets, showing the impact of expert number, Top-K, LoRA rank, and edge density.

Analysis of the model's hyperparameters revealed several critical insights:

  • The Power of Sparsity: GraphLoRA performs best with an edge density of only 10%. This suggests that a fully connected network of experts isn't necessary; a small amount of shared context is sufficient for effective collaboration.
  • Efficiency through Collaboration: GraphLoRA achieved its best results with a LoRA rank as low as 2. Competing models often require ranks of 4 or 8. This indicates that a smarter, graph-based router allows for much smaller, more efficient experts.

Conclusion and Future Work

GraphLoRA demonstrates that load imbalance in MoE is not just a distribution problem but a communication problem. By facilitating expert collaboration via a graph router and specialized distribution losses, the framework achieves superior accuracy and stability with fewer trainable parameters.

Future research may explore dynamic graph learning, where the model learns the optimal connections between experts during training rather than relying on a static topology. Additionally, scaling these graph-based approaches to models exceeding 100B parameters remains an important next step for the field.

Summary updated

Abstract

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method that has been widely adopted in various downstream applications of LLMs. Together with the Mixture-of-Expert (MoE) technique, fine-tuning approaches have shown remarkable improvements in model capability. However, the coordination of multiple experts in existing studies solely relies on the weights assigned by the simple router function. Lack of communication and collaboration among experts exacerbate the instability of LLMs due to the imbalance load problem of MoE. To address this issue, we propose a novel MoE graph-based LLM fine-tuning framework GraphLoRA, in which a graph router function is designed to capture the collaboration signals among experts by graph neural networks (GNNs). GraphLoRA enables all experts to understand input knowledge and share information from neighbor experts by aggregating operations. Besides, to enhance each expert's capability and their collaborations, we design two novel coordination strategies: the Poisson distribution-based distinction strategy and the Normal distribution-based load balance strategy. Extensive experiments on four real-world datasets demonstrate the effectiveness of our GraphLoRA in parameter-efficient fine-tuning of LLMs, showing the benefits of facilitating collaborations of multiple experts in the graph router of GraphLoRA.
BibTeX
@misc{bai2024graphlora,
  author = {Ting Bai and Yue Yu and Le Huang and Zenan Xu and Zhe Zhao and Chuan Shi},
  title = {{GraphLoRA: Empowering LLMs Fine-Tuning via Graph Collaboration of MoE}},
  year = {2024},
  eprint = {2412.16216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.16216},
}
Figure