Introduction
The training of state-of-the-art Large Language Models (LLMs) has increasingly become a resource-intensive endeavor, often restricted to entities with massive GPU clusters. While Mixture-of-Experts (MoE) architectures offer a path toward scaling by activating only a subset of parameters per token, the total memory footprint remains a significant barrier. Collaborative training approaches like Federated Learning offer a potential solution but are frequently hampered by high local hardware requirements and vulnerabilities to data-reconstruction attacks.
This paper introduces Privacy-preserving Collaborative MoE (PC-MoE), a framework designed to enable multiple parties to collaboratively train large-scale MoE models. PC-MoE addresses the dual challenges of hardware constraints and privacy by distributing expert parameters and leveraging the inherent sparsity of MoE routing as a natural defense mechanism.
The PC-MoE Framework
The core of PC-MoE is a distributed architecture that partitions the model across multiple participants. Unlike traditional federated learning where each party maintains a full copy of the model, PC-MoE distributes the workload:
- Distributed Architecture: Each participant (Party A, B, ..., N) hosts a local "backbone" consisting of non-expert layers and a unique subset (shard) of the global expert pool.
- Sparse Routing: A local gating layer directs tokens to specific experts. If the required expert resides on a different party's server, only the necessary hidden activations are transmitted, rather than raw data or full model weights.
- Local Updates: Training proceeds without global weight averaging. Each party updates only the parameters they own—their backbone and their specific experts—based on gradients transmitted through the sparse paths.
Experimental Evaluation
The researchers evaluated PC-MoE across seven diverse benchmarks, including ARC-C, OpenBookQA, SuperGLUE, MMLU-Redux, AGIEval, BBH, and MedQA. These tasks cover a wide range of capabilities from general knowledge to specialized medical reasoning.
Performance and Utility
A critical requirement for any collaborative training framework is maintaining the performance of a centralized baseline. As shown in the experimental results, PC-MoE achieves accuracy levels that are nearly identical to, and sometimes exceed, those of a centralized model where all data is pooled.
Memory Efficiency
By amortizing the memory cost of heavy expert parameters across the federation, PC-MoE significantly lowers the entry barrier for participants. The framework achieved a reduction in peak GPU RAM usage of nearly 70% compared to standard training methods. This allows groups with mid-tier hardware to collectively train models that would otherwise require high-end H100 clusters.
Privacy Analysis
PC-MoE utilizes the sparsity of MoE routing as a privacy shield. Because any single token only interacts with a tiny fraction of the total experts, an adversary hosting a subset of experts only receives a highly sparsified signal. In "gradient inversion" attack tests, the framework proved robust; reconstructed text snippets yielded extremely low ROUGE scores (e.g., ROUGE-2 scores of 0.000 across multiple datasets), indicating that the recovered data was essentially unintelligible.
Conclusion
PC-MoE represents a significant step toward democratizing the training of trillion-parameter LLMs. By combining the efficiency of MoE architectures with a privacy-preserving distributed training protocol, it allows multiple parties to pool resources without compromising data security or model utility. Future work will focus on optimizing the sequential training bottlenecks and exploring the feasibility of full-scale pre-training in decentralized environments.
Summary updated