TL;DR
Training multiple deep neural networks (DNNs) on a single GPU is challenging due to memory limitations. Zico introduces a novel system that intelligently shares GPU memory between concurrent training jobs, significantly improving efficiency and throughput, especially when memory is scarce.
What Problem Does This Paper Address?
The increasing complexity of deep neural networks (DNNs) has led to a surge in their use for various applications. Training these models, however, is computationally intensive and requires powerful hardware, particularly GPUs. As GPUs become a bottleneck, sharing them among multiple training jobs is crucial for efficient resource utilization. Existing methods for sharing GPUs either temporally (time-slicing) or spatially (dividing compute units) struggle with the rapidly increasing memory demands of concurrent DNN training. This "memory bloating" can limit the effectiveness of these sharing techniques, leading to underutilization or even system instability. Zico aims to solve this by efficiently managing and sharing GPU memory across concurrent training jobs, reducing overall system memory consumption and maximizing throughput.
Methods
Zico introduces a system with two core components: a scrooge scheduler and a memory manager.
- Scrooge Scheduler: This component orchestrates the execution of concurrent training jobs. It monitors the memory usage patterns of each job in real-time, predicting future memory demands. Based on these predictions and a defined memory budget, it intelligently decides when to start each job's iteration, aiming to overlap memory allocation (forward pass) with deallocation (backward pass) to maximize sharing. It uses a feedback-driven approach to refine its scheduling decisions based on observed memory lifespans.
- Memory Manager: This component acts as a unified memory allocator for the GPU. It organizes GPU memory into "regions" based on tensor types (e.g., feature maps, gradient maps, temporary tensors). This segregation helps prevent interference between different types of data. It manages both sharable (global) and non-sharable (job-local) memory pools. To ensure correctness, Zico uses CUDA events to track actual GPU kernel completion before deallocating memory, preventing corruption that can occur with CPU-based deallocation in asynchronous GPU processing.
Datasets
The paper does not explicitly mention specific datasets used for training the DNN models. However, it evaluates Zico on six well-known DNN benchmarks: NASNet, ResNet-110, ResNet-50, GNMT, RHN, and BERT. These models represent a range of tasks and memory usage patterns, allowing for a comprehensive evaluation of Zico's performance across different workloads.
Key Insights and Ideas
- Memory Bloating is the Bottleneck: The paper highlights that the primary limitation in concurrent DNN training isn't just compute, but the significant and often unpredictable memory demands of feature maps, which grow with model complexity and concurrent execution.
- Coordinated Sharing is Key: Simply dividing GPU resources isn't enough. Zico's success lies in its ability to coordinate the timing of memory allocation and deallocation across jobs to create opportunities for sharing.
- Feature Maps are the Target: Feature maps, due to their long lifespan and large size, are identified as the primary candidates for memory sharing. Zico's strategy focuses on efficiently managing these.
- Asynchronous GPU Processing Requires Careful Tracking: The inherent asynchrony between CPU and GPU necessitates precise tracking of GPU memory usage (via CUDA events) to avoid memory corruption when sharing.
- Workload Variability is Handled: Zico's scrooge scheduler is designed to adapt to diverse memory usage patterns across different DNN models, making it broadly applicable.
Key Results
- Significant Throughput Improvements: Zico outperforms traditional spatial sharing (MPS) by up to 8.3x and temporal sharing by 1.6x on average, especially when dealing with high memory footprint scenarios and non-identical models.
- Effective Memory Management: Zico successfully reduces system-wide memory consumption, enabling concurrent training even when individual jobs would normally exceed GPU memory capacity.
- Robustness Across Models: Zico demonstrates strong performance across a variety of DNN models with different memory usage patterns, from lean (GNMT) to beefy (ResNet-50).
- Minimal Overhead: The scrooge scheduler's scheduling decisions have negligible overhead, ensuring that the benefits of memory sharing are not offset by scheduling costs.
Limitations and Future Work
- Current Design for Two Jobs: The current design of Zico primarily focuses on co-locating two training jobs. The authors suggest that organizing jobs into pairs and scheduling each pair independently is a feasible approach for more than two jobs, which they leave as future work.
- Balancing Individual Job Throughput: While Zico improves aggregate throughput, it doesn't guarantee fair individual job throughput. In scenarios where one job has low compute demand, it might be scheduled less frequently to maximize overall GPU utilization, potentially impacting the performance of that specific job. Balancing individual job throughput alongside aggregate throughput is identified as future work.
- Dynamic Memory Budget Adaptation: The paper demonstrates Zico's ability to adapt to changes in the memory budget, but further exploration of more complex dynamic scenarios could be beneficial.
Summary updated