Abstract
Training large-scale deep neural networks (DNNs) is computationally expensive. While cloud providers offer preemptible "spot" instances at significant discounts, their unpredictable reclamation poses a major challenge for continuous training. Existing reactive systems rely on checkpointing or redundancy, which often struggle with high I/O overhead or resource waste under frequent preemptions. We present Parcae, a proactive system that optimizes for a novel metric called liveput—the expected throughput across potential preemption scenarios. By predicting resource availability and performing tiered live migrations, Parcae achieves up to 10x better performance than reactive baselines, making spot-instance training both robust and cost-effective.
Introduction
Training modern DNNs like GPT-3 is staggeringly expensive, often costing millions of dollars in compute. Cloud providers offer "preemptible" or "spot" instances at a 90% discount, but these instances can be reclaimed at any time with minimal notice. Existing systems for spot-instance training are primarily reactive. They either use checkpointing (saving state to disk and rolling back after a failure) or redundancy (running the same math on multiple nodes). Both approaches have significant flaws: checkpointing incurs massive I/O overhead as models grow, and redundancy wastes expensive GPU cycles. When preemptions are frequent, these systems often spend more time recovering than training, sometimes making zero progress on large-scale models.
Methods
Parcae shifts the paradigm from reactive recovery to proactive adaptation through several novel components:
- Liveput Metric: Instead of just optimizing for raw throughput (samples per second), Parcae optimizes for liveput—the expected throughput across various possible preemption scenarios. It essentially asks: "How fast can I train, and how much of that work will survive if $N$ nodes disappear?"
- Statistical Availability Predictor: Rather than trying to predict exactly which GPU will be reclaimed, Parcae uses an ARIMA (Auto-Regressive Integrated Moving Average) model to forecast the number of available instances in the near future.
- Liveput Optimizer: A dynamic programming algorithm searches the space of parallelization strategies (data vs. pipeline parallelism) to find the configuration that maximizes liveput based on the forecast. A key insight is that different parallelization strategies have different "fragility"; Parcae may choose a slightly slower, shorter pipeline because it is more likely to survive an upcoming "storm" of preemptions.
- Tiered Live Migration: When a preemption is predicted or announced, Parcae uses the cloud's "grace period" (typically 30 seconds) to migrate model states between GPUs. It uses three strategies—intra-stage, inter-stage, and pipeline migration—to rebalance the workload without hitting slow persistent storage.
- ParcaePS: A specialized parameter server keeps up-to-date checkpoints in cheap on-demand CPU DRAM, providing a fast "safety net" for rare cases where migration fails.
Evaluation
The researchers evaluated Parcae using popular DNN architectures including ResNet-152, BERT-Large, and GPT-3 (up to 6.7B parameters) using real-world preemption traces from AWS.
Key Results
- Massive Speedups: Parcae outperformed SOTA systems like Varuna (checkpoint-based) and Bamboo (redundancy-based) by up to 10x in high-preemption scenarios. As shown in the performance traces, Parcae maintains a steady training pace while reactive systems stall during recovery phases.
- Cost Efficiency: Training with Parcae was 3.24x cheaper than using standard on-demand instances while maintaining near-optimal performance.
- Scalability: For the 6.7B parameter GPT-3 model, existing systems failed to make any progress under frequent preemptions, while Parcae achieved 89% of the performance of an "ideal" system with perfect future knowledge.
- Convergence: The authors verified that Parcae's dynamic reconfigurations do not harm the final accuracy of the model, as Stochastic Gradient Descent (SGD) is naturally robust to data reordering.
Conclusion
Parcae demonstrates that proactive resource management and the optimization of "liveput" can unlock the massive cost savings of spot instances for large-scale DNN training. While the system currently relies on short grace periods provided by cloud providers, its tiered migration and ParcaePS safety net provide a robust framework for future developments in preemptible computing. Future work may extend this approach to include more complex operator-splitting strategies and more advanced ML-based availability predictors.
Summary updated