Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances

Jiangfei Duan, Ziang Song, Xupeng Miao, Dahua Lin, Harry Xu, Minjia Zhang, Zhihao Jia

Abstract

Training large-scale deep neural networks (DNNs) is computationally expensive. While cloud providers offer preemptible "spot" instances at significant discounts, their unpredictable reclamation poses a major challenge for continuous training. Existing reactive systems rely on checkpointing or redundancy, which often struggle with high I/O overhead or resource waste under frequent preemptions. We present Parcae, a proactive system that optimizes for a novel metric called liveput—the expected throughput across potential preemption scenarios. By predicting resource availability and performing tiered live migrations, Parcae achieves up to 10x better performance than reactive baselines, making spot-instance training both robust and cost-effective.

Introduction

Training modern DNNs like GPT-3 is staggeringly expensive, often costing millions of dollars in compute. Cloud providers offer "preemptible" or "spot" instances at a 90% discount, but these instances can be reclaimed at any time with minimal notice. Existing systems for spot-instance training are primarily reactive. They either use checkpointing (saving state to disk and rolling back after a failure) or redundancy (running the same math on multiple nodes). Both approaches have significant flaws: checkpointing incurs massive I/O overhead as models grow, and redundancy wastes expensive GPU cycles. When preemptions are frequent, these systems often spend more time recovering than training, sometimes making zero progress on large-scale models.

Methods

Parcae shifts the paradigm from reactive recovery to proactive adaptation through several novel components:

  • Liveput Metric: Instead of just optimizing for raw throughput (samples per second), Parcae optimizes for liveput—the expected throughput across various possible preemption scenarios. It essentially asks: "How fast can I train, and how much of that work will survive if $N$ nodes disappear?"
  • Statistical Availability Predictor: Rather than trying to predict exactly which GPU will be reclaimed, Parcae uses an ARIMA (Auto-Regressive Integrated Moving Average) model to forecast the number of available instances in the near future.
  • Liveput Optimizer: A dynamic programming algorithm searches the space of parallelization strategies (data vs. pipeline parallelism) to find the configuration that maximizes liveput based on the forecast. A key insight is that different parallelization strategies have different "fragility"; Parcae may choose a slightly slower, shorter pipeline because it is more likely to survive an upcoming "storm" of preemptions.
  • Tiered Live Migration: When a preemption is predicted or announced, Parcae uses the cloud's "grace period" (typically 30 seconds) to migrate model states between GPUs. It uses three strategies—intra-stage, inter-stage, and pipeline migration—to rebalance the workload without hitting slow persistent storage.
  • ParcaePS: A specialized parameter server keeps up-to-date checkpoints in cheap on-demand CPU DRAM, providing a fast "safety net" for rare cases where migration fails.

Evaluation

The researchers evaluated Parcae using popular DNN architectures including ResNet-152, BERT-Large, and GPT-3 (up to 6.7B parameters) using real-world preemption traces from AWS.

Performance comparison showing the number of mini-batches processed over time. Parcae significantly outperforms reactive systems like Varuna and Bamboo, approaching ideal performance.

Key Results

  • Massive Speedups: Parcae outperformed SOTA systems like Varuna (checkpoint-based) and Bamboo (redundancy-based) by up to 10x in high-preemption scenarios. As shown in the performance traces, Parcae maintains a steady training pace while reactive systems stall during recovery phases.
  • Cost Efficiency: Training with Parcae was 3.24x cheaper than using standard on-demand instances while maintaining near-optimal performance.
  • Scalability: For the 6.7B parameter GPT-3 model, existing systems failed to make any progress under frequent preemptions, while Parcae achieved 89% of the performance of an "ideal" system with perfect future knowledge.
  • Convergence: The authors verified that Parcae's dynamic reconfigurations do not harm the final accuracy of the model, as Stochastic Gradient Descent (SGD) is naturally robust to data reordering.

Conclusion

Parcae demonstrates that proactive resource management and the optimization of "liveput" can unlock the massive cost savings of spot instances for large-scale DNN training. While the system currently relies on short grace periods provided by cloud providers, its tiered migration and ParcaePS safety net provide a robust framework for future developments in preemptible computing. Future work may extend this approach to include more complex operator-splitting strategies and more advanced ML-based availability predictors.

Summary updated

Abstract

Deep neural networks (DNNs) are becoming progressively large and costly to train. This paper aims to reduce DNN training costs by leveraging preemptible instances on modern clouds, which can be allocated at a much lower price when idle but may be preempted by the cloud provider at any time. Prior work that supports DNN training on preemptive instances employs a reactive approach to handling instance preemptions and allocations after their occurrence, which only achieves limited performance and scalability. We present Parcae, a system that enables cheap, fast, and scalable DNN training on preemptible instances by proactively adjusting the parallelization strategy of a DNN training job to adapt to predicted resource changes before instance preemptions and allocations really happen, which significantly reduces the cost of handling these events. Parcae optimizes liveput, a novel metric that measures the expected training throughput of a DNN job under various possible preemption scenarios. Compared to existing reactive, throughputoptimized systems, Parcae’s proactive, live-optimized solution considers both the throughput of a job and its robustness under preemptions. To optimize liveput, Parcae supports lightweight instance migration and uses an availability predictor to forecast future preemptions. It then uses a liveput optimizer to discover an optimal strategy to parallelize DNN training under predicted preemptions. We evaluate Parcae on a variety of DNNs and preemption traces and show that Parcae outperforms existing spot-instance DNN training systems by up to 10×. More importantly, Parcae achieves near-optimal performance for training large DNNs under frequent preemptions, in which case existing approaches cannot make any progress.
BibTeX
@inproceedings{duan2024parcae,
  author = {Jiangfei Duan and Ziang Song and Xupeng Miao and Dahua Lin and Harry Xu and Minjia Zhang and Zhihao Jia},
  title = {{Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances}},
  booktitle = {NSDI'24},
  year = {2024},
  url = {https://openreview.net/forum?id=A1j235Vzcu},
}
Figure