TL;DR
ThunderServe is a high-performance LLM serving system specifically designed for the "messy" reality of heterogeneous cloud environments. By splitting the prefill and decode phases across different GPU types and utilizing a novel two-level hierarchical scheduler, it achieves up to a 2.1× increase in throughput and a 2.5× reduction in latency compared to state-of-the-art systems within the same budget.
What Problem Does This Paper Address?
Serving Large Language Models (LLMs) is notoriously expensive and hardware-intensive. While most research assumes "ideal" environments—homogeneous clusters of top-tier GPUs like NVIDIA A100s connected by high-speed NVLink—the reality of cloud computing is different. Developers often face a "GPU shortage" and must settle for heterogeneous clusters containing various GPU generations (e.g., A100s, A6000s, and 3090Tis) connected by relatively slow Ethernet or PCIe.
Furthermore, LLM inference has two distinct personalities:
- Prefill Phase: Computationally intensive (compute-bound) as it processes the initial prompt.
- Decode Phase: Memory-bandwidth intensive (memory-bound) as it generates tokens one by one.
Existing systems often fail to account for how these different phases interact with diverse hardware and fluctuating workloads. If a workload shifts from short prompts (conversation) to long prompts (coding), a static deployment plan becomes highly inefficient.
Methods
The authors introduce ThunderServe, which employs several novel strategies to bridge the gap between heterogeneous hardware and LLM requirements:
- Phase Splitting: Instead of running the entire inference on one GPU, ThunderServe separates prefill and decode tasks onto different "model serving groups." This allows the system to match the workload to the hardware (e.g., using high-compute GPUs for prefill and high-bandwidth GPUs for decoding).
- Two-Level Hierarchical Scheduling:
- Upper Level: Uses a Tabu search algorithm to partition available GPUs into groups and designate them as either prefill or decode replicas.
- Lower Level: Optimizes the internal parallel configuration (Tensor and Pipeline Parallelism) for each group and uses a Two-Stage Transportation Problem (TSTP) formulation to orchestrate request routing between prefill and decode replicas.
- Lightweight Re-scheduling: To handle "workload shifts" or node failures, ThunderServe can swap the roles of existing replicas (e.g., turning a prefill group into a decode group) in seconds. This avoids the massive overhead of re-calculating a plan from scratch and reloading model weights, which can take minutes.
- KV Cache Compression: To mitigate the slow interconnects in cloud environments, the system quantizes the Key-Value (KV) cache to 4-bit precision for transmission between nodes, dequantizing it only for computation to maintain model accuracy.
Key Insights and Ideas
- Hardware-Phase Matching: The "Aha!" moment is that an older or "lesser" GPU might actually be better for a specific phase. For example, an NVIDIA 3090Ti has excellent memory bandwidth, making it a cost-effective choice for the decode phase, even if it lacks the raw TFLOPS of an A100.
- The Communication Tax: In the cloud, the network is the bottleneck. The authors found that without KV cache compression and smart orchestration, the cost of moving data between prefill and decode replicas can negate the benefits of phase splitting.
- Role Flipping over Reloading: By simply changing the designation of a GPU group rather than its configuration, the system can adapt to dynamic traffic (like a sudden surge in long-form coding requests) with almost zero downtime.
Key Results
- Efficiency: Given the same price budget, ThunderServe outperformed HexGen (a heterogeneous baseline) by 1.5× to 1.8× in throughput.
- Cost-Effectiveness: Compared to industry standards like vLLM and DistServe running on homogeneous in-house clusters, ThunderServe achieved up to a 2.1× throughput increase and a 1.5× average reduction in latency deadlines.
- Resilience: The lightweight re-scheduling mechanism finished in roughly 13 seconds, compared to over 150 seconds for a full system restart and parameter reload.
- Accuracy: 4-bit KV cache compression reduced communication overhead by up to 30% while keeping the accuracy drop under 2% across all tested tasks.
Limitations and Future Work
The authors acknowledge a few areas for further exploration:
- Static Budgeting: The current scheduler assumes a fixed pool of GPUs. Future iterations could integrate with cloud auto-scaling to dynamically rent or release GPUs based on demand.
- Quantization Limits: While 4-bit quantization worked well, the system uses a "one-shot" approach. Exploring quantization-aware training or more sophisticated compression could potentially push this to 2-bit without significant quality loss.
- State Management: The paper focuses on throughput and latency but doesn't deeply explore the complexities of state persistence or recovery for the KV cache if a node fails mid-generation.
Summary updated