Scaling Neural Networks Beyond Single Device Memory

This direction is merged into another direction: model parallelism. Its last version stays readable and citable here.

Version 1 of 136 papers39 conceptsfrozen 2026-10-04

In the early days of distributed deep learning, the primary bottleneck was data. To speed up training, researchers used data parallelism: they replicated the entire model on multiple GPUs and fed each one a different slice of the dataset. However, as models scaled from millions to hundreds of billions of parameters, a new wall was hit. The model itself became too large to fit into the memory (VRAM) of a single accelerator.

This shift from "Big Data" to "Big Model" problems necessitated a move toward model parallelism. Instead of replicating the model, researchers began partitioning the model’s weights, activations, and optimizer states across a cluster of devices.

The Limits of Replication

Standard frameworks like ChainerMN by Takuya Akiba et al. achieved high efficiency by using synchronous data parallelism. But as Fuxun Yu et al. point out in their survey on Large-Scale Deep Learning Serving, the "one model, one device" approach breaks down when a model reaches the trillion-parameter scale.

When a single layer or embedding table exceeds the local memory of a GPU, the system must shard the architecture. As Seunghak Lee and Jin Kyu Kim argue in their work on STRADS, the goal of model parallelism is to ensure that the memory requirement per machine actually decreases as you add more nodes to the cluster.

Strategies for Partitioning

There are three primary ways to slice a model across a cluster:

  1. Width (Tensor) Parallelism: This involves splitting individual layers—such as large weight matrices—across multiple GPUs. For the CPM (Chinese Pre-trained Model), Xu Han and Zhengyan Zhang partitioned the architecture along the width dimension to reduce data transfer bottlenecks.
  2. Depth (Pipeline) Parallelism: Here, different layers of the network are placed on different devices. Naveen Arivazhagan and Ankur Bapna utilized pipeline parallelism to train a universal translation model with 1.3 billion parameters, allowing the model to process 103 languages simultaneously.
  3. Optimizer and Gradient Sharding: Even if the weights fit, the "hidden" costs of training—gradients and optimizer states—can consume 10x more memory than the weights themselves. The OpenBMB ecosystem, led by Guoyang Zeng, uses this form of sharding to train 170-billion-parameter models on hardware that would normally be limited to models 50 times smaller.

Scaling Across Domains

While large-scale modeling began in ASR with researchers like Ciprian Chelba distributing n-gram counts across clusters, the technique has moved into specialized fields:

  • Clinical NLP: Xi Yang and Aokun Chen developed GatorTron, an 8.9-billion parameter model for electronic health records, by segmenting the encoder across multiple GPUs.
  • Computer Vision: Mostafa Dehghani and Yi Tay scaled vision transformers to 22 billion parameters (ViT-22B) using model-parallel sharding, proving that vision models can benefit from the same massive scaling laws as LLMs.
  • Speech Recognition: Yu Zhang and Ruoming Pang used automated pipeline parallelism to train BigSSL, an 8-billion parameter Conformer model, on a million hours of audio.

The Coordination Challenge

The primary difficulty in model parallelism is communication overhead. When a model is split, the GPUs must constantly exchange intermediate activations and gradients.

To mitigate this, frameworks like STRADS use intelligent scheduling to prioritize which parameters to update, while Yaoliang Yu and Yi Zhou developed msPG, a model-parallel algorithm that ensures convergence even in "stale" systems where nodes are slightly out of sync.

By treating the cluster as a single, unified fabric of memory rather than a collection of isolated devices, these techniques allow researchers to build models whose intelligence is limited by the size of the cluster, not the size of a single chip.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scaling_neural_networks_beyond_single_de_706640bc,
  title = {Scaling Neural Networks Beyond Single Device Memory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-neural-networks-beyond-single-device-memory/rdr_706640bc235d434e9913468d2dda81a3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_neural_networks_beyond_single_de_706640bc_v1,
  title = {Scaling Neural Networks Beyond Single Device Memory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-neural-networks-beyond-single-device-memory/rdr_706640bc235d434e9913468d2dda81a3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}