Heterogeneous Federated Learning with Scalable Server Mixture-of-Experts

Jingang Jiang, Yanzhao Chen, Xiangyang Liu, Haiqi Jiang, Chenyou FanIJCAI 2025 · DOI

TL;DR

This paper introduces Fed-MoE, a novel federated learning framework that addresses the challenge of deploying large models on resource-constrained edge devices. By employing a Mixture-of-Experts (MoE) architecture on the server and compact models on clients, Fed-MoE enables efficient aggregation of diverse client knowledge into a powerful, unified server model, achieving state-of-the-art results on various vision and language tasks.

What Problem Does This Paper Address?

Federated Learning (FL) allows distributed model training while preserving data privacy. However, a major hurdle is the requirement for identical model architectures across all clients for parameter averaging. This becomes problematic when deploying large, computationally intensive models (like LLMs or large visual models) on edge devices with limited resources. Classical FL struggles to accommodate this heterogeneity. The paper tackles this by proposing an asymmetric FL approach where clients use compact models, and the server utilizes a much larger, more sophisticated Mixture-of-Experts (MoE) model. This allows the server to leverage the collective intelligence of many clients without burdening them with large model computations.

Methods

The core of Fed-MoE is its asymmetric architecture and a three-stage iterative update process:

  • Asymmetric Architecture: Clients train compact, single-expert models. The server hosts a large MoE model, comprising a "main expert" that always activates and several "routed experts" that are selectively activated by a gating module. Crucially, each server expert shares the same architecture as the client models.
  • Three-Stage Update Process:
    1. Stage A: Local Training and Upload: Clients train their compact models on local data and upload them to the server.
  • Stage B: Server Experts and Gate Update: This is the most novel part.
  • Probing Client Responses: The server uses a small reserved dataset to get responses from client models.
  • Server-Client Correlation: A relevance score is calculated between each server expert and each client model.
  • Server Expert Update: Server experts are updated using a weighted FedAvg, where weights are derived from the server-client correlation. This dynamically dispatches client expertise to relevant server experts.
  • Server Gate Update: The MoE gating module is updated to better route inputs to the most suitable experts, incorporating a "Gating Entropy" loss to encourage diversification of expert specializations.
  • Stage C: Client Synchronization: Updated server experts and the gating module are synchronized back to clients, allowing them to refine their local models.

This process allows the server MoE to learn from diverse client data distributions, even when they are non-IID, by intelligently fusing client knowledge.

Datasets

The paper evaluates Fed-MoE on several standard benchmarks:

  • Vision Tasks:
    • FEMNIST: A federated version of MNIST, partitioned by writing styles.
  • CIFAR-10: Used with a Dirichlet distribution to simulate Non-IID scenarios.
  • Language Tasks:
    • SENT-140: A sentiment classification task.
  • YELP: A 5-way review star classification task.

The authors also utilize a small "server reserved dataset" for training the server's gating module, which is distinct from the client data.

Key Insights and Ideas

  • Asymmetric FL is Viable: The paper demonstrates that breaking the symmetry in model size and complexity between clients and servers is not only possible but highly effective for large-scale FL.
  • Expertise Dispatching is Key: The core innovation lies in the dynamic relevance calculation and weighted aggregation of client models to server experts. This allows the server to intelligently fuse knowledge, rather than just averaging parameters.
  • Gating Entropy for Specialization: The introduction of Gating Entropy loss is crucial for encouraging server experts to specialize in distinct data segments, which is particularly beneficial for Non-IID data.
  • Main Expert for General Knowledge: The inclusion of a main expert that always activates provides a baseline of general knowledge, complementing the specialized routed experts.
  • Scalability to Large Models: Fed-MoE shows promise for training billion-parameter models in a federated setting, a significant step towards democratizing access to powerful AI.

Key Results

Fed-MoE consistently outperforms existing FL and MoE baselines across various datasets and client configurations:

  • Vision Tasks: Achieves significant accuracy improvements, e.g., over 10% higher than FedAvg and FedJETs on FEMNIST (M=50), and 5-7% higher than FedMix on CIFAR-10.
  • Language Tasks: Outperforms baselines like FedMix and FedProx by 1.7-7.45% on SENT-140 and 1.2-2.1% on Yelp.
  • Communication Efficiency: While Fed-MoE has a larger server model, its communication cost is significantly lower than some MoE baselines (e.g., FedMix) due to efficient client-to-server knowledge transfer.
  • Robustness to Non-IID: The framework demonstrates strong performance even under highly Non-IID data distributions.
  • Ablation Studies:
    • Using Top-1 routed expert during inference yields the best accuracy while minimizing computation.
  • Dynamic update of server experts and gates is critical, outperforming static MoE approaches.
  • A moderate weight for Gating Entropy loss strikes a balance between specificity and versatility.

Limitations and Future Work

  • Server Reserved Data: The performance is sensitive to the size of the server reserved dataset, suggesting that a reasonably sized dataset is necessary for effective gating module training.
  • Hyperparameter Tuning: While the paper explores some hyperparameters (like Gating Entropy weight), further tuning might be required for optimal performance across diverse scenarios.
  • Future Work: The authors suggest extending the applicability to tasks like image and text classification with billion-scale MoE systems, which they have already demonstrated. Further exploration of dynamic expert selection and more sophisticated client-server matching mechanisms could be future directions.

Summary updated

BibTeX
@inproceedings{jiang2025heterogeneous,
  author = {Jingang Jiang and Yanzhao Chen and Xiangyang Liu and Haiqi Jiang and Chenyou Fan},
  title = {{Heterogeneous Federated Learning with Scalable Server Mixture-of-Experts}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/610},
}
Figure