AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, Ion Stoica

Introduction

Deep learning (DL) models are growing exponentially in size, making production serving increasingly complex. Traditionally, model parallelism—the technique of partitioning a model across multiple hardware accelerators—has been treated as a last resort, used only when a model is too large to fit into the memory of a single GPU. AlpaServe reimagines model parallelism as a proactive tool for "statistical multiplexing." By partitioning even relatively small models across multiple GPUs, the system allows multiple models to share a common resource pool. This approach drastically reduces queuing delays during traffic bursts and improves request processing rates by up to 10x while maintaining strict latency targets.

The Challenges of Modern DL Serving

Most existing serving systems follow a "simple placement" strategy where one model is assigned to one GPU. This paradigm faces two significant hurdles in production environments:

  1. Bursty and Skewed Workloads: Real-world traffic is rarely steady. Production traces show that request spikes can reach 50x the average rate. In a one-model-per-GPU setup, a burst for a specific model leads to long queues and Service Level Objective (SLO) violations, even if adjacent GPUs hosting other models are idle.
  2. Memory and Resource Constraints: Large-scale models, such as GPT-3, cannot fit on a single GPU. While model parallelism is used to solve this, it is typically applied statically and conservatively, missing opportunities to optimize for the latency-sensitive nature of serving.

AlpaServe addresses this "statistical multiplexing gap" by enabling multiple models to share a pool of GPUs, allowing idle resources to help absorb bursts from any model in the collection.

Methodology

AlpaServe introduces a novel serving framework that treats model parallelism as a tunable parameter to optimize for SLO attainment. The system is built on several key technical pillars:

Model Parallelism Taxonomy

The system navigates the trade-offs between two primary forms of parallelism:

  • Intra-op Parallelism: Splitting a single layer across GPUs to reduce the execution latency of an individual request.
  • Inter-op Parallelism: Pipelining different layers across GPUs to increase the overall throughput of the system.

Auto-Parallelization for Inference

Building on the Alpa compiler, AlpaServe optimizes model partitions specifically for the inference phase. Unlike training, which focuses on maximizing total throughput, AlpaServe’s compiler minimizes the "maximal stage latency" to ensure the fastest possible response time for individual requests.

Simulator-Guided Placement

Determining the optimal grouping of models and GPUs is a complex combinatorial problem. AlpaServe employs a greedy, two-level algorithm guided by a discrete-event simulator. It categorizes models into "buckets" by size and searches for a placement strategy that maximizes the percentage of requests finished before their deadline.

Centralized Runtime

A centralized controller tracks queue lengths across GPU groups in real-time. It dispatches incoming requests to the group with the shortest queue, ensuring that the computational load is balanced dynamically across the cluster.

Evaluation

The authors evaluated AlpaServe using production-grade traces and representative model architectures on a cluster of 64 NVIDIA Tesla V100 GPUs (16GB).

  • Workload Traces: The evaluation used Microsoft Azure Function traces from 2019 (MAF1, characterized as steady and dense) and 2021 (MAF2, characterized as highly bursty and skewed).
  • Models: The study included the BERT family (ranging from 1.3B to 104B parameters) and GShard Mixture of Experts (MoE) models (ranging from 1.3B to 5.3B parameters).

Key Insights and Results

The Parallelism Trade-off

Model parallelism introduces communication overhead, which can increase the raw execution time of a single request. However, AlpaServe demonstrates that this is often a price worth paying. By processing a burst of requests across multiple GPUs simultaneously, the system significantly decreases the time requests spend waiting in a queue. For example, a 10% execution overhead is beneficial if it reduces average completion time by 25% during a traffic spike.

Performance Gains

Compared to state-of-the-art baselines like Clockwork++ and Selective Replication, AlpaServe achieved:

  • 10x Higher Throughput: The system processes significantly more requests while staying within latency constraints.
  • Superior Burst Tolerance: AlpaServe handles up to 6x more burstiness (coefficient of variation in traffic) than traditional methods.
  • Resource Efficiency: To meet the same SLO, AlpaServe requires up to 2.3x fewer GPUs than competing systems.
  • High SLO Attainment: In production traces, AlpaServe consistently met latency targets for over 99% of requests, even when deadlines were 2.5x more stringent than what baselines could handle.

Limitations and Future Work

While AlpaServe represents a significant step forward, several areas remain for future exploration. The placement algorithm currently assumes that arrival processes can be approximated from historical traces; extreme "black swan" events may still challenge static placement. Additionally, the current evaluation focuses on non-autoregressive models. Extending these techniques to generative models (like GPT-4), which require complex KV-cache management and have unknown request lengths, is a priority. Finally, the authors suggest exploring how AlpaServe could handle multi-tenant serving of parameter-efficient fine-tuning (PEFT) modules, such as LoRA adapters.

Summary updated

BibTeX
@inproceedings{li2023alpaserve,
  author = {Zhuohan Li and Lianmin Zheng and Yinmin Zhong and Vincent Liu and Ying Sheng and Xin Jin and Yanping Huang and Zhifeng Chen and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
  title = {{AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving}},
  booktitle = {OSDI},
  year = {2023},
  url = {https://openreview.net/forum?id=Spng4BtMbA},
}
Figure