TL;DR
This paper introduces StreamBox, a novel lightweight GPU sandbox for serverless inference workflows. By leveraging GPU streams and optimizing memory management and communication, StreamBox significantly reduces GPU memory footprint and cold start latency, enabling more efficient and cost-effective serverless GPU inference.
What Problem Does This Paper Address?
The increasing demand for Deep Neural Network (DNN) inference in intelligent applications has led to the adoption of serverless computing for its elasticity and cost-efficiency. However, existing serverless inference systems often rely on monolithic GPU runtimes (like CUDA contexts) for each function. This approach is inefficient for short-lived, fine-grained inference tasks, leading to:
- High startup latency: Each function requires a heavy GPU runtime initialization.
- Large memory footprint: Redundant GPU runtimes consume significant memory, limiting the number of concurrent functions.
- Expensive inter-function communication: Data often needs to be copied between CPU and GPU memory, slowing down workflows.
- Underutilization of GPU resources: Monolithic runtimes are too heavy for the actual inference computation, leaving much of the GPU idle.
The paper aims to address these limitations by proposing a more efficient and lightweight approach to GPU resource management in serverless inference.
Methods
StreamBox introduces several key innovations to create a lightweight GPU sandbox:
- GPU Streams as Sandboxes: Instead of isolating functions with separate, heavy GPU runtimes, StreamBox allows functions within a workflow to share a single GPU runtime using GPU streams. Streams enable concurrent kernel execution and shared memory access, similar to threads in a process.
- Auto-scaling Memory Pool: StreamBox implements a fine-grained, auto-scaling memory pool. It allocates and recycles memory at the layer granularity of DNNs, reducing waste. It also uses offline profiling to build memory usage models, making the pool resilient to varying function demands and avoiding over-allocation.
- User-Transparent Communication Framework: StreamBox provides unified communication APIs that abstract away the complexities of inter-function communication. It adaptively switches between efficient communication methods (direct memory access, NVLink, RPC) based on function placement and utilizes an "elastic communication store" to manage intermediate data in GPU memory, avoiding competition.
- Fine-grained PCIe Bandwidth Sharing: To address the bottleneck of PCIe bandwidth, StreamBox partitions data into smaller blocks and uses an I/O daemon for global scheduling. It also employs techniques like right-sizing data blocks (2MB), preemptive batch transfers, and shared pinned memory buffers to improve transfer efficiency.
Datasets
The paper evaluates StreamBox using real-world workloads derived from the Azure Function trace [4]. They also use three representative DNN inference workflows: Image Processing, Traffic, and Social Media. No new benchmarks were introduced; instead, they focused on evaluating their system against existing ones using these established traces and workflows.
Key Insights and Ideas
- Monolithic GPU runtimes are the bottleneck: The paper convincingly demonstrates that the overhead of separate GPU runtimes is a major impediment to efficient serverless GPU inference, far outweighing the actual computation cost.
- GPU streams are a natural fit for serverless: By treating GPU streams as the fundamental unit for function execution within a shared runtime, StreamBox unlocks significant efficiency gains.
- Fine-grained memory management is crucial: Simply allocating large memory pools is wasteful. StreamBox's layer-level, auto-scaling approach is key to reducing memory footprint and cost.
- Communication optimization is non-trivial: Efficiently sharing data between functions, especially within the same GPU, requires careful management of shared memory and avoiding unnecessary data copies. StreamBox's elastic communication store and adaptive movement policies are vital here.
- PCIe bandwidth is a critical resource: The paper highlights how fine-grained I/O scheduling and data block management are necessary to prevent one function from monopolizing the PCIe link and to ensure fair sharing.
Key Results
StreamBox demonstrates significant improvements over state-of-the-art serverless inference systems:
- Reduced GPU Memory Footprint: Up to 82% reduction compared to INFless and Astraea.
- Improved Throughput: Up to 6.7X higher throughput compared to INFless and Astraea.
- Lower Latency: Significantly reduced end-to-end latency, with startup latency reduced from over 5 seconds to as low as 5ms.
- Efficient Resource Utilization: StreamBox allows for higher concurrency of functions on a single GPU.
Limitations and Future Work
- Multi-tenancy: While StreamBox can be used in multi-tenant scenarios by starting multiple API proxies, its effectiveness in reducing GPU runtime redundancy diminishes when many users share a single GPU. However, it still mitigates data redundancy and cold starts for functions within the same workflow.
- Workflow-aware Scheduling: The paper mentions that they employ existing workflow-aware function scheduling but suggests that further integration of function scheduling could lead to even better intermediate data management.
- GPU Virtualization: The paper notes that GPU virtualization (like MIG) is orthogonal to their work and could potentially be combined with StreamBox for enhanced isolation in multi-tenant scenarios.
- Other GPU Types: While the paper focuses on NVIDIA GPUs, it mentions that AMD GPUs also support streams, suggesting potential for broader applicability.
Summary updated