Fine-Grained GPU Sharing Primitives for Deep Learning Applications

Peifeng Yu, Mosharaf ChowdhuryMLSys 2020 · OpenReview

TL;DR

Modern GPUs are powerful but inefficiently shared by deep learning (DL) applications due to coarse-grained allocation. This paper introduces Salus, a system that enables fine-grained GPU sharing through fast job switching and memory lane abstraction, significantly improving utilization and reducing job completion times.

What Problem Does This Paper Address?

Deep learning (DL) applications have exploded in popularity, heavily relying on GPUs for their parallel processing capabilities. However, current GPU resource management is "all-or-nothing," meaning a single GPU can only be allocated to one application at a time. This coarse-grained allocation leads to two major inefficiencies:

  1. Scheduling Inflexibility: Cluster managers struggle to efficiently schedule DL jobs. Policies like First-In, First-Out (FIFO) lead to head-of-line (HOL) blocking, where a long-running job can delay many shorter ones. Preemption, a common technique for other resources, is costly on GPUs due to the overhead of saving and restoring job states.
  2. GPU Underutilization: Many DL jobs, especially those with large batch sizes or fluctuating workloads, don't fully utilize a GPU's resources (both compute and memory). This leaves valuable GPU capacity idle, which could otherwise be used by other applications. This is particularly problematic for hyper-parameter tuning and inference serving, where resource demands can be bursty or low on average.

Methods

Salus tackles these problems by introducing two core GPU sharing primitives:

  1. Fast Job Switching: Salus enables rapid switching between DL jobs on a GPU. This is achieved by leveraging the observation that DL jobs have predictable memory usage patterns with distinct "persistent" (model parameters) and "ephemeral" (intermediate computations) components. Salus keeps the persistent memory on the GPU and only switches the ephemeral allocations, drastically reducing the overhead compared to traditional checkpointing/restoring methods. This allows for efficient time-sharing and preemption, enabling policies like Shortest Remaining Time First (SRTF).
  2. Memory Sharing via GPU Lane Abstraction: Salus divides GPU memory into "persistent" and "ephemeral" regions. The ephemeral region is further segmented into "lanes." Each lane can host multiple DL jobs, with execution serialized within a lane using GPU streams. This allows for packing multiple jobs onto a single GPU, improving utilization. The system also incorporates an "auto defragmentation" mechanism at iteration boundaries to manage memory fragmentation.

These primitives are integrated into a consolidated execution service that DL frameworks can access via a framework-specific adaptor, making Salus transparent to the end-user.

Datasets

The paper does not explicitly mention specific public datasets used for training. Instead, it evaluates Salus using a collection of popular DL workloads and models, including those from the official TensorFlow CNN benchmarks and other widely used models in various fields. They created 45 distinct workloads by varying batch sizes for 15 different models. For inference, they selected 42 DL inference jobs from 14 different models.

Key Insights and Ideas

  • DL Memory Patterns Enable Fine-Grained Sharing: The key insight is that the predictable, spatiotemporal nature of DL memory usage (persistent model vs. ephemeral intermediate data) is not a bug but a feature that can be exploited for efficient GPU sharing.
  • Iteration-Level Granularity is Optimal: While kernel-level switching was explored, it introduced too much overhead. Iteration-level switching, enabled by Salus's primitives, strikes a good balance between utilization and efficiency.
  • Persistent Memory is the Key to Fast Switching: By keeping model parameters (persistent memory) on the GPU and only managing ephemeral memory, Salus achieves significantly faster job switching than traditional methods that involve moving entire job states.
  • GPU Lanes for Packing: The GPU lane abstraction effectively partitions memory, allowing multiple jobs to share a GPU without interfering with each other's critical memory needs, thus enabling efficient packing.
  • Deadlock Avoidance is Crucial: The paper highlights a potential deadlock scenario in progressive memory allocation and explains how Salus's iteration-based approach avoids this.

Key Results

Salus demonstrates significant improvements across various DL workloads:

  • Long-Running Training: Improves average job completion time (JCT) by 3.19x compared to FIFO using the SRTF scheduling policy.
  • Hyper-Parameter Tuning: Increases GPU utilization by 2.38x by allowing multiple tuning jobs to run concurrently.
  • Inference: Achieves 42x GPU utilization improvement over not sharing the GPU and 7x over NVIDIA MPS, while maintaining low latency.
  • Overhead: Salus introduces minimal per-iteration training time overhead (less than 10% for most CNN models).
  • Comparison with Existing Solutions: Salus achieves comparable performance to NVIDIA MPS for training jobs but offers much greater flexibility.

Limitations and Future Work

  • Scheduling Policy Optimization: While Salus provides the primitives, the paper acknowledges that the "best" scheduling algorithm for DL jobs on a shared GPU is an open research question.
  • Multi-GPU and Multi-Machine Extension: The current work focuses on single-GPU sharing. Future work includes extending Salus to multiple GPUs on a single machine and across multiple machines using RDMA.
  • CPU Computation Optimization: Salus currently redirects CPU computations to its own execution engine, which is not yet heavily optimized. Further optimization in this area could reduce overhead for models with significant CPU components.
  • Automatic Lane Assignment: The paper notes that finding the optimal number of lanes and their assignment is an area for further exploration.

Summary updated

BibTeX
@inproceedings{yu2020fine,
  author = {Peifeng Yu and Mosharaf Chowdhury},
  title = {{Fine-Grained GPU Sharing Primitives for Deep Learning Applications}},
  booktitle = {MLSys},
  year = {2020},
  url = {https://openreview.net/forum?id=-bjEozXE_LjM},
}
Figure