Abstract Federated learning (FL) is a collaborative machine learning approach that enables multiple clients to train models without sharing their private data. With the rise of deep learning, large-scale models have garnered significant attention due to their exceptional performance. However, a key challenge in FL is the limitation imposed by clients with constrained computational and communication resources, which hampers the deployment of these large models. The Mixture of Experts (MoE) architecture addresses this challenge with its sparse activation property, which reduces computational workload and communication demands during inference and updates. Additionally, MoE facilitates better personalization by allowing each expert to specialize in different subsets of the data distribution. To alleviate the communication burdens between the server and clients, we propose FedMoE-DA, a new FL model training framework that leverages the MoE architecture and incorporates a novel domain-aware, fine-grained aggregation strategy to enhance the robustness, personalizability, and communication efficiency simultaneously. Specifically, the correlation between both intra-client expert models and inter-client data heterogeneity is exploited. Moreover, we utilize peer-to-peer (P2P) communication between clients for selective expert model synchronization, thus significantly reducing the server-client transmissions. Experiments demonstrate that our FedMoE-DA achieves excellent performance while reducing the communication pressure on the server.
Introduction
In the world of Federated Learning (FL), a constant tension exists between the desire for massive, high-performance models and the practical limitations of "edge" clients—such as smartphones and IoT devices—which often possess limited computational power and unstable network connections. Furthermore, data heterogeneity (non-IID data) across clients makes a universal global model perform poorly for individual users.
To address these challenges, the FedMoE-DA framework combines the efficiency of Mixture of Experts (MoE) with a "domain-aware" peer-to-peer (P2P) aggregation strategy. This approach enables high-performance personalization while significantly reducing the communication bottleneck at the central server.
The FedMoE-DA Framework
The FedMoE-DA architecture decomposes the model into three distinct components to balance global robustness with local personalization:
- Shared Embedding Model: A shallow layer trained via standard FedAvg. It captures universal low-level features (e.g., edges in images) shared across all clients.
- Private Gating Network: A local "router" that determines which specific experts should be activated for a given input.
- Private Experts: A pool of specialized sub-networks that handle specific data domains.
As shown in the workflow above, the framework utilizes the gating network parameters as "proxies" for local data distributions. Instead of transmitting large expert models to the central server, clients only upload these compact gating proxies. The server then uses these proxies to identify clients with similar data distributions.
Domain-Aware Fine-grained Aggregation
The core innovation of FedMoE-DA lies in its aggregation strategy. The central server uses the uploaded gating proxies to compute a similarity matrix. If two clients have similar gating weights, it indicates their data distributions likely overlap.
Rather than relying on the server to distribute experts, FedMoE-DA leverages Peer-to-Peer (P2P) communication. Based on the similarity matrix provided by the server, clients selectively request the most relevant experts from their peers. This "fine-grained" aggregation ensures that clients only download knowledge that is beneficial to their specific data domain, reducing unnecessary communication and noise. To maintain efficiency, the similarity matrix is updated periodically rather than every round, allowing clients to operate with "stale" but still effective routing information.
Experimental Results
The researchers evaluated FedMoE-DA on several benchmarks, including MNIST, EMNIST, CIFAR-10, and SVHN, using Dirichlet distributions to simulate varying degrees of data heterogeneity.
Key findings include:
- Superior Accuracy: On CIFAR-10 with high heterogeneity ($\alpha = 0.1$), FedMoE-DA achieved 81.03% accuracy, significantly outperforming baselines like Scaffold (58.67%) and FedRoD (80.08%).
- Scalability with Experts: As illustrated in the figures above, increasing the number of experts ($P$) retrieved via P2P generally improves both the final test accuracy and the speed of convergence. The framework maintains high performance without the "noise" of irrelevant knowledge from dissimilar clients.
- Communication Efficiency: By offloading expert exchange to P2P and using sparse activation, the framework drastically reduces the traffic load on the central server. The authors also noted that using a "frozen" pre-trained embedding model can further minimize central model aggregation requirements.
Conclusion
FedMoE-DA demonstrates that sparse MoE architectures, when paired with domain-aware P2P aggregation, can effectively scale federated learning to large models on resource-constrained devices. By using gating networks as signatures of local data, the framework achieves high personalization without compromising communication efficiency. Future work may explore the use of Knowledge Distillation to allow for heterogeneous expert architectures and further investigate the privacy implications of P2P model exchange.
Summary updated