No article text was recorded for this version.
model parallelism
Version 1 of 1157 papers186 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Primitives for Dynamic Big Model Parallelism
- Blog: ZeRO: memory optimizations toward training trillion parameter models
- Blog: LightLDA: Big Topic Models on Modest Compute Clusters
- Blog: Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- Blog: P4SGD: Programmable Switch Enhanced Model-Parallel Training on Generalized Linear Models on Distributed FPGAs
- Blog: PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- Blog: Petuum: A Framework for Iterative-Convergent Distributed ML
- Blog: Pretraining Billion-scale Geospatial Foundational Models on Frontier
- Blog: A Model Parallel Proximal Stochastic Gradient Algorithm for Partially Asynchronous Systems
- Blog: On Convergence of Model Parallel Proximal Gradient Algorithm for Stale Synchronous Parallel System
- Blog: Primitives for Dynamic Big Model Parallelism
- Blog: Extreme Stochastic Variational Inference: Distributed and Asynchronous
- Blog: Pretraining Billion-Scale Geospatial Foundational Models on Frontier
- Blog: Model-parallel Fourier neural operators as learned surrogates for large-scale parametric PDEs
- Blog: Fully distributed EM for very large datasets
- Blog: On Model Parallelization and Scheduling Strategies for Distributed Machine Learning
- Blog: Model-Parallel Inference for Big Topic Models
- Blog: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Blog: ZeRO: Memory Optimization Towards Training A Trillion Parameter Models
- Blog: Local Critic Training for Model-Parallel Learning of Deep Neural Networks
- Blog: Distributed Proximal Gradient Algorithm for Partially Asynchronous Computer Clusters
- Blog: Whack-a-Chip: The Futility of Hardware-Centric Export Controls
- Blog: Distributed Function Minimization in Apache Spark
- Blog: A Data and Model-Parallel, Distributed and Scalable Framework for Training of Deep Networks in Apache Spark
- Blog: Optimizing Distributed Training on Frontier for Large Language Models
- Blog: Fugue: Slow-Worker-Agnostic Distributed Learning for Big Models on Big Data
- Blog: Survey on Efficient Training of Large Neural Networks
- Blog: Benchmark Assessment for Optimization Andacceleration Libraries in Deep Learning-Based Models
- Blog: A simple linear algebra identity to optimize Large-Scale Neural Network Quantum States
- Blog: Fog Intelligence for Network Anomaly Detection
- Blog: Larger-Scale Transformers for Multilingual Masked Language Modeling
- Blog: Tesseract: Parallelize the Tensor Parallelism Efficiently
- Blog: OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery
- Blog: Large Scale Distributed Deep Networks
- Blog: Distributed Deep Reinforcement Learning: A Survey and a Multi-player Multi-agent Learning Toolbox
- Blog: Scaling Up Models and Data with t5x and seqio
- Blog: Whale: Scaling Deep Learning Model Training to the Trillions.
- Blog: Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
- Blog: Parallel Training for Deep Stacking Networks
- Blog: Scaling Back-propagation by Parallel Scan Algorithm
- Blog: Familia: A Configurable Topic Modeling Framework for Industrial Text Engineering
- Blog: Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training
- Blog: PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
- Blog: HyperCluster: Decentralized Large Language Model Inference over Peer-to-Peer Wireless Networks
- Blog: Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
- Blog: Tech Report A Variational HEM Algorithm for Clustering Hidden Markov Models
- Blog: EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models
- Blog: CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X
- Blog: Declarative Recursive Computation on an RDBMS: or, Why You Should Use a Database For Distributed Machine Learning
- Blog: Scalable Artificial Intelligence for Science: Perspectives, Methods and Exemplars
- Blog: OLMo: Accelerating the Science of Language Models
- Blog: Fog Intelligence for Network Anomaly Detection
- Blog: LOFT: Finding Lottery Tickets through Filter-wise Training
- Blog: Deep Neural Networks: Another Tool for Multimedia Computing
- Blog: OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
- Blog: Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
- Blog: On Convergence of Model Parallel Proximal Gradient Algorithm for Stale Synchronous Parallel System
- Blog: GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- Blog: Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques, and Tools
- Blog: GraphScale: A Framework to Enable Machine Learning over Billion-node Graphs
Showing 60 of 157 papers.
History & citation
Current · Permanent link
- Scaling Neural Networks Beyond Single Device Memory merged into this direction (67% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_model_parallelism_bbc7c799,
title = {model parallelism},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/model-parallelism/rdr_bbc7c79938244e4582bef2e6dde99a5c},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}This exact version (version 1, frozen text):
@misc{lacuna_model_parallelism_bbc7c799_v1,
title = {model parallelism},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/model-parallelism/rdr_bbc7c79938244e4582bef2e6dde99a5c/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}