Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts

Max Ryabinin, Anton Gusev

Introduction

Training massive neural networks like GPT-3 is prohibitively expensive, often requiring specialized supercomputers that cost hundreds of millions of dollars. This financial barrier restricts state-of-the-art AI research to a few well-funded organizations. While "volunteer computing" (e.g., Folding@home) has successfully aggregated the power of consumer PCs for other scientific domains, traditional distributed deep learning methods fail in this environment. Data and model parallelism require high-speed, highly reliable interconnects to constantly sync massive amounts of data—conditions impossible to meet on consumer internet connections characterized by high latency, low bandwidth, and frequent node failures.

To bridge this gap, researchers propose Learning@home, a novel paradigm that crowdsources the training of large models using everyday consumer-grade PCs.

The Learning@home Architecture

The Learning@home infrastructure relies on a decentralized architecture designed specifically to thrive in unreliable networks. It is built on three main technical pillars:

Decentralized network topology showing workers, neighbors, and expert routing
  • Decentralized Mixture-of-Experts (DMoE): Instead of standard dense layers that require syncing the entire model across all nodes, Learning@home utilizes a Mixture-of-Experts architecture. Individual "experts" (independent sub-networks) are hosted on different volunteer nodes. A lightweight gating function evaluates an input and selects only the top-$k$ experts to process it. This drastically reduces communication overhead since a node only talks to a few peers per forward pass.
  • Distributed Hash Tables (DHT): To manage the dynamic pool of volunteers without relying on a centralized, vulnerable server, the system uses a Kademlia DHT. This allows the network to route data dynamically and seamlessly handle nodes joining or leaving at any time.
  • Asynchronous Training with Gradient Checkpointing: Trainers process batches asynchronously to keep hardware utilized despite network delays. They also use gradient checkpointing to avoid storing intermediate activations while waiting for remote experts to return results.
Step-by-step training process in DMoE, illustrating DHT routing, forward/backward passes, and handling of disconnected experts

Key Insights

  • MoE is Uniquely Suited for Crowdsourcing: Because only a small fraction of experts are activated for any given input, the massive communication bottleneck of traditional distributed training is bypassed.
  • Natural Staleness Mitigation: In asynchronous training, gradients usually get "stale" (out of sync) and hurt convergence. However, in DMoE, concurrent batches are highly likely to be routed to completely different experts. This means their weight updates rarely conflict, naturally mitigating the stale gradient problem.
  • Node Failures Act as Regularization: If a volunteer's computer drops offline, the gating function simply ignores that expert and renormalizes the weights across the remaining selected experts. This acts remarkably like Dropout, forcing the network to learn redundant, robust representations.

Experimental Results

The authors tested Learning@home across simulated high-latency environments and standard datasets (MNIST, WikiText-2):

  • Throughput Under Latency: In a simulated high-latency environment (~200ms delay), DMoE maintained a throughput of ~300 samples/s for a feed-forward network. In stark contrast, standard pipeline model parallelism collapsed to just ~7 samples/s.
  • Robust Convergence: On the MNIST task, DMoE successfully converged even with an artificial 1-second network latency and a 10% node failure rate.
  • Language Modeling: When training a Transformer-XL on WikiText-2, the DMoE model matched the baseline's performance. Surprisingly, introducing a 10% node failure rate actually improved the validation perplexity, acting as a strong regularizer against overfitting the limited training data.
  • Scalability: The DHT routing scales logarithmically. Finding the top-4 experts took only ~764ms in a simulated network of 10,000 nodes, proving the bookkeeping can handle massive crowds.

Limitations and Future Work

While promising, the decentralized nature of Learning@home introduces new challenges. The system is vulnerable to malicious actors; a hacked node could intentionally return incorrect outputs or poisoned gradients (e.g., NaNs) to sabotage the entire training process. Additionally, the system currently saturates 100-200 Mbps connections, which could strain consumer ISPs if heavily adopted.

Future work will focus on implementing statistical techniques (like ignoring outlier gradients) to defend against malicious nodes, exploring secure DHT variations to prevent Sybil attacks, and optimizing the protocol to reduce the network burden. The authors are continuing to develop Learning@home as an open-source project to democratize large-scale AI research.

Summary updated

Abstract

Many recent breakthroughs in deep learning were achieved by training increasingly larger models on massive datasets. However, training such models can be prohibitively expensive. For instance, the cluster used to train GPT-3 costs over $250 million. As a result, most researchers cannot afford to train state of the art models and contribute to their development. Hypothetically, a researcher could crowdsource the training of large neural networks with thousands of regular PCs provided by volunteers. The raw computing power of a hundred thousand $2500 desktops dwarfs that of a $250M server pod, but one cannot utilize that power efficiently with conventional distributed training methods. In this work, we propose Learning@home: a novel neural network training paradigm designed to handle large amounts of poorly connected participants. We analyze the performance, reliability, and architectural constraints of this paradigm and compare it against existing distributed training techniques.
BibTeX
@inproceedings{ryabinin2020towards,
  author = {Max Ryabinin and Anton Gusev},
  title = {{Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=zdUoYIIGTKD},
}
Figure