Abstract
Deep learning is experiencing a rise in foundation models that are expected to lead in various fields. The massive number of parameters necessitates the use of tensor model parallelism (TMP) in foundation model training. However, TMP requires frequent communication operations which significantly reduces the training efficiency. In this paper, we present Oases, an automated TMP method with overlapped communication to accelerate foundation model training. Oases proposes a fine-grained training schedule to maximize overlapping communication and computation operations that have data dependence. Additionally, we design the Oases planner that searches for the best model parallel strategy to achieve further accelerations. Unlike existing methods, Oases planner is specifically tailored to model the cost of overlapped communication-computation operations. We evaluate Oases on various model settings and train environments, and compare Oases to four stat-of-the-art implementations. Experimental results demonstrate that Oases achieves speedups of 1.01--1.48X over the fastest baseline, and speedups of up to 1.9X over Megatron-LM.
TL;DR
Oases is an automated training framework designed to make large-scale model training efficient on "commodity" hardware (like RTX 3090 GPUs) that lacks high-speed interconnects. By aggressively overlapping communication with computation across different training passes and using an ILP-based planner to optimize model partitioning, Oases achieves up to a 1.95× speedup over the industry-standard Megatron-LM.
What Problem Does This Paper Address?
Training "Foundation Models" (like LLaMA or GPT) typically requires massive clusters of high-end GPUs (like A100s or H100s) connected by ultra-fast NVLink or InfiniBand. Most researchers, however, only have access to "commodity" servers. On these cheaper systems, the communication bandwidth is a massive bottleneck.
When models grow too large for a single GPU's memory, developers use Tensor Model Parallelism (TMP) to split individual layers across multiple GPUs. However, TMP is "chatty"—it requires frequent synchronization (AllReduce operations) that forces the GPU to sit idle while waiting for data. Existing systems either don't overlap this communication well or require manual, sub-optimal tuning of how the model is split. Oases aims to "democratize" large-scale training by making TMP efficient even on low-bandwidth networks.
Methods
Oases introduces two primary innovations to solve the communication bottleneck:
- Fine-Grained Overlapping Schedule:
- Cross-pass Scheduling: Traditional training treats the forward, recomputation, and backward passes as distinct blocks with "barriers" between them. Oases breaks these barriers by splitting data into sub-batches. While one sub-batch is performing computation, the system asynchronously handles the communication for the next sub-batch, even if it belongs to a different training pass.
* **Redundant Communication Removal:** The authors mathematically proved that during the "recomputation" phase (used to save memory), the synchronization operations (AllReduce) are often redundant for gradient calculation. Oases skips these, reducing the total communication volume.
- The Oases Planner:
- Instead of using a uniform "one-size-fits-all" strategy for every layer, Oases uses an Integer Linear Programming (ILP) solver to find the optimal partitioning strategy for each specific layer.
- The planner uses a sophisticated cost model that accounts for the time saved by overlapping and the "resharding" penalty (the cost of moving data when adjacent layers use different levels of parallelism).
Datasets
As a systems-focused paper, the authors evaluated Oases using several prominent Transformer-based model architectures rather than specific training datasets. The models included:
- BERT (Large and 1.3B versions)
- GPT (8.3B, 18.4B, 39.1B, and 7.5B versions)
- LLaMA (7B and 65B versions)
- GPT-NeoX
Evaluation was performed on two 8-node commodity clusters: one with RTX 3090s using NVLink and one without.
Key Insights and Ideas
- Breaking the Pass Barrier: The most significant "aha moment" is that the strict separation of forward and backward passes is an artificial constraint. By interleaving sub-batches across these passes, you can find computation to hide almost any communication latency.
- Math Saves Bandwidth: Realizing that the gradients of an AllReduce's inputs are identical to the gradients of its outputs allows Oases to safely delete communication steps during recomputation without affecting the model's convergence or accuracy.
- Heterogeneity is Key: Even in a "homogeneous" cluster, the optimal parallelism strategy might vary layer-by-layer. A layer with high computation density can afford more communication than a "thin" layer. The Oases Planner automates this complex trade-off.
Key Results
- Speedups: Oases achieved speedups of 1.01–1.48× over the fastest existing baselines (like Alpa and Merak) and up to 1.95× over Megatron-LM.
- Device Utilization: The framework improved GPU utilization by 1.17–2.18×. In scenarios where Megatron-LM saw GPUs idling for 64% of the time due to communication, Oases kept them productive.
- Scalability: The system demonstrated strong "weak scaling," meaning it maintains high efficiency as more GPUs are added to the cluster.
- Planner Efficiency: The ILP planner is extremely fast, finding optimal strategies for complex models in less than half a second.
Limitations and Future Work
- Architecture Specificity: Currently, Oases is specifically tailored and evaluated for Transformer-based architectures. While the principles apply elsewhere, the implementation is not yet "plug-and-play" for non-Transformer models.
- Arithmetic Density Trade-off: Splitting batches into smaller sub-batches to enable overlapping can slightly reduce the "arithmetic density" (how efficiently the GPU handles a single operation). The authors note a performance loss of up to 20% in raw computation speed, though this is more than offset by the communication savings.
- Future Directions: The authors plan to extend Oases to support other parallel methods (like sequence parallelism) and a wider variety of deep learning architectures.
Summary updated