PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs

Ze Yu Zhang, Bolin Ding, Bryan Kian Hsiang LowarXiv 2025 · DOI · arXiv

Introduction

The training of state-of-the-art Large Language Models (LLMs) has increasingly become a resource-intensive endeavor, often restricted to entities with massive GPU clusters. While Mixture-of-Experts (MoE) architectures offer a path toward scaling by activating only a subset of parameters per token, the total memory footprint remains a significant barrier. Collaborative training approaches like Federated Learning offer a potential solution but are frequently hampered by high local hardware requirements and vulnerabilities to data-reconstruction attacks.

This paper introduces Privacy-preserving Collaborative MoE (PC-MoE), a framework designed to enable multiple parties to collaboratively train large-scale MoE models. PC-MoE addresses the dual challenges of hardware constraints and privacy by distributing expert parameters and leveraging the inherent sparsity of MoE routing as a natural defense mechanism.

The PC-MoE Framework

The core of PC-MoE is a distributed architecture that partitions the model across multiple participants. Unlike traditional federated learning where each party maintains a full copy of the model, PC-MoE distributes the workload:

  1. Distributed Architecture: Each participant (Party A, B, ..., N) hosts a local "backbone" consisting of non-expert layers and a unique subset (shard) of the global expert pool.
  2. Sparse Routing: A local gating layer directs tokens to specific experts. If the required expert resides on a different party's server, only the necessary hidden activations are transmitted, rather than raw data or full model weights.
  3. Local Updates: Training proceeds without global weight averaging. Each party updates only the parameters they own—their backbone and their specific experts—based on gradients transmitted through the sparse paths.
Overview of the PC-MoE architecture showing distributed gating and expert layers across multiple parties.

Experimental Evaluation

The researchers evaluated PC-MoE across seven diverse benchmarks, including ARC-C, OpenBookQA, SuperGLUE, MMLU-Redux, AGIEval, BBH, and MedQA. These tasks cover a wide range of capabilities from general knowledge to specialized medical reasoning.

Performance and Utility

A critical requirement for any collaborative training framework is maintaining the performance of a centralized baseline. As shown in the experimental results, PC-MoE achieves accuracy levels that are nearly identical to, and sometimes exceed, those of a centralized model where all data is pooled.

Comparison of accuracy across various benchmarks for different training baselines and PC-MoE.

Memory Efficiency

By amortizing the memory cost of heavy expert parameters across the federation, PC-MoE significantly lowers the entry barrier for participants. The framework achieved a reduction in peak GPU RAM usage of nearly 70% compared to standard training methods. This allows groups with mid-tier hardware to collectively train models that would otherwise require high-end H100 clusters.

Privacy Analysis

PC-MoE utilizes the sparsity of MoE routing as a privacy shield. Because any single token only interacts with a tiny fraction of the total experts, an adversary hosting a subset of experts only receives a highly sparsified signal. In "gradient inversion" attack tests, the framework proved robust; reconstructed text snippets yielded extremely low ROUGE scores (e.g., ROUGE-2 scores of 0.000 across multiple datasets), indicating that the recovered data was essentially unintelligible.

Conclusion

PC-MoE represents a significant step toward democratizing the training of trillion-parameter LLMs. By combining the efficiency of MoE architectures with a privacy-preserving distributed training protocol, it allows multiple parties to pool resources without compromising data security or model utility. Future work will focus on optimizing the sequential training bottlenecks and exploring the feasibility of full-scale pre-training in decentralized environments.

Summary updated

Abstract

Mixture-of-Experts (MoE) has been gaining popularity due to its successful adaptation to large language models (LLMs). In this work, we introduce Privacy-preserving Collaborative Mixture-of-Experts (PC-MoE), which leverages the sparsity of the MoE architecture for memory-efficient decentralized collaborative LLM training, enabling multiple parties with limited GPU-memory and data resources to collectively train more capable LLMs than they could achieve individually. At the same time, this approach protects training data privacy of each participant by keeping training data, as well as parts of the forward pass signal and gradients locally within each party. By design, PC-MoE synergistically combines the strengths of distributed computation with strong confidentiality assurances. Unlike most privacy-preserving schemes, which pay for confidentiality with lower task accuracy, our framework breaks that trade-off: across seven popular LLM benchmarks, it almost matches (and sometimes exceeds) the performance and convergence rate of a fully centralized model, enjoys near 70% peak GPU RAM reduction, while being fully robust against reconstruction attacks.
BibTeX
@misc{zhang2025pc,
  author = {Ze Yu Zhang and Bolin Ding and Bryan Kian Hsiang Low},
  title = {{PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs}},
  year = {2025},
  eprint = {2506.02965},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.02965},
}
Figure