FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models

Panlong Wu, Kangshuo Li, Ting Wang, Fangxin Wang

Abstract

Foundation models have shown great success in natural language processing, computer vision, and multimodal tasks. FMs have a large number of model parameters, thus requiring a substantial amount of data to help optimize the model during the training. Federated learning has revolutionized machine learning by enabling collaborative learning from decentralized data while still preserving the data privacy of clients. Despite the great benefits foundation models can have empowered by federated learning, they face severe computation, communication, and statistical challenges. In this paper, we propose a novel two-stage federated learning algorithm called FedMS. A global expert is trained in the first stage and a local expert is trained in the second stage to provide better personalization. We construct a Mixture of Foundation Models (MoFM) with these two experts and design a gate neural network with an inserted gate adapter that joins the aggregation every communication round in the second stage. To further adapt to edge computing scenarios with limited computational resources, we design a novel Sparsely Activated LoRA (SAL) algorithm that freezes the pre-trained foundation model parameters inserts low-rank adaptation matrices into transformer blocks and activates them progressively during the training. We employ extensive experiments to verify the effectiveness of FedMS, results show that FedMS outperforms other SOTA baselines by up to 55.25% in default settings.

Introduction

Foundation Models (FMs) like CLIP or GPT have redefined the state-of-the-art in various domains. However, their massive parameter counts pose significant challenges for deployment on resource-constrained edge devices, particularly within the Federated Learning (FL) paradigm. FL aims to train models on decentralized data without compromising privacy, but it faces three major hurdles when paired with FMs:

  1. Communication Bottlenecks: The sheer size of FM updates makes frequent transmission over mobile networks impractical.
  2. Computation Constraints: Edge devices often lack the hardware necessary for full fine-tuning of large models.
  3. Data Heterogeneity: Non-IID (Independent and Identically Distributed) data across clients often leads to poor performance for a single global model.

To address these, we introduce FedMS (Federated Learning with Mixture of Sparsely Activated Foundation Models), a framework designed to bring the power of FMs to the edge efficiently.

Methodology: The FedMS Framework

FedMS introduces two primary innovations: the Mixture of Foundation Models (MoFM) and Sparsely Activated LoRA (SAL).

Mixture of Foundation Models (MoFM)

FedMS operates in two distinct stages to balance global knowledge with local specialization. In Stage 1, clients collaboratively train a Global Expert using Low-Rank Adaptation (LoRA). This stage focuses on capturing general features across the entire federation. In Stage 2, each client initializes a Local Expert from the global one and trains it alongside a Gating Network.

The final inference is a weighted combination of the global and local experts' outputs. To maintain synchronization without high overhead, a lightweight Gate Adapter is inserted into the gating network; only this adapter is aggregated at the server during Stage 2.

The two-stage FedMS architecture: Stage 1 trains a global expert, while Stage 2 introduces a local expert and a gating mechanism for personalization.

Sparsely Activated LoRA (SAL)

To mitigate computational costs, we propose SAL, a progressive parameter activation strategy. Rather than activating all trainable LoRA layers simultaneously, SAL uses a Capability Queue to monitor local accuracy. When performance plateaus, a controller activates additional LoRA layers in higher transformer blocks. This "curriculum learning" approach ensures that devices only utilize the computational capacity required for the current stage of learning, significantly reducing energy consumption.

Experimental Evaluation

We evaluated FedMS against several state-of-the-art baselines, including PromptFL and LayerFreeze Fine-Tuning (LFFT), across three diverse datasets:

  • Food101: 101 categories of food images.
  • UCF101: Human action recognition.
  • EuroSAT: Satellite imagery for land use classification.

Performance and Efficiency

FedMS demonstrated superior performance across all benchmarks, particularly in highly non-IID settings. It achieved an accuracy improvement of up to 55.25% over competing methods.

Accuracy comparison of FedMS against baselines (FT, LFFT, PromptFL) across Food101, UCF101, and EuroSAT datasets under varying degrees of data heterogeneity (non-iid alpha).

Beyond accuracy, the efficiency gains were substantial. FedMS reduced communication time by 99.95% compared to full fine-tuning. In a simulated low-bandwidth environment (0.1 MB/s), a single communication round was completed in just 3.08 seconds, whereas full fine-tuning would have required over 1.6 hours.

Discussion and Conclusion

The success of FedMS stems from its ability to leverage a frozen global backbone while allowing for lightweight local adaptation. The "Mixture of Experts" approach provides a robust safety net of general knowledge while enabling the model to specialize in local nuances. Furthermore, the SAL strategy proves that full model capacity is not necessary from the start of training, offering a path toward more sustainable AI on edge devices.

While currently validated on Vision-Language Models like CLIP, future work will explore the application of FedMS to Large Language Models (LLMs) and investigate its performance in dynamic environments with rapidly shifting data distributions.

Summary updated

Abstract

Foundation models have shown great success in natural language processing, computer vision, and multimodal tasks. FMs have a large number of model parameters, thus requiring a substantial amount of data to help optimize the model during the training. Federated learning has revolutionized machine learning by enabling collaborative learning from decentralized data while still preserving the data privacy of clients. Despite the great benefits foundation models can have empowered by federated learning, they face severe computation, communication, and statistical challenges. In this paper, we propose a novel two-stage federated learning algorithm called FedMS. A global expert is trained in the first stage and a local expert is trained in the second stage to provide better personalization. We construct a Mixture of Foundation Models (MoFM) with these two experts and design a gate neural network with an inserted gate adapter that joins the aggregation every communication round in the second stage. To further adapt to edge computing scenarios with limited computational resources, we design a novel Sparsely Activated LoRA (SAL) algorithm that freezes the pre-trained foundation model parameters inserts low-rank adaptation matrices into transformer blocks and activates them progressively during the training. We employ extensive experiments to verify the effectiveness of FedMS, results show that FedMS outperforms other SOTA baselines by up to 55.25% in default settings.
BibTeX
@misc{wu2023fedms,
  author = {Panlong Wu and Kangshuo Li and Ting Wang and Fangxin Wang},
  title = {{FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models}},
  year = {2023},
  eprint = {2312.15926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.15926},
}
Figure