ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments

Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin Cui, Ana Klimovic, Eiko Yoneki

TL;DR

ThunderServe is a high-performance LLM serving system specifically designed for the "messy" reality of heterogeneous cloud environments. By splitting the prefill and decode phases across different GPU types and utilizing a novel two-level hierarchical scheduler, it achieves up to a 2.1× increase in throughput and a 2.5× reduction in latency compared to state-of-the-art systems within the same budget.

What Problem Does This Paper Address?

Serving Large Language Models (LLMs) is notoriously expensive and hardware-intensive. While most research assumes "ideal" environments—homogeneous clusters of top-tier GPUs like NVIDIA A100s connected by high-speed NVLink—the reality of cloud computing is different. Developers often face a "GPU shortage" and must settle for heterogeneous clusters containing various GPU generations (e.g., A100s, A6000s, and 3090Tis) connected by relatively slow Ethernet or PCIe.

Furthermore, LLM inference has two distinct personalities:

  1. Prefill Phase: Computationally intensive (compute-bound) as it processes the initial prompt.
  2. Decode Phase: Memory-bandwidth intensive (memory-bound) as it generates tokens one by one.

Existing systems often fail to account for how these different phases interact with diverse hardware and fluctuating workloads. If a workload shifts from short prompts (conversation) to long prompts (coding), a static deployment plan becomes highly inefficient.

Methods

The authors introduce ThunderServe, which employs several novel strategies to bridge the gap between heterogeneous hardware and LLM requirements:

The hierarchical scheduling framework of ThunderServe, from cluster information to phase orchestration.
  • Phase Splitting: Instead of running the entire inference on one GPU, ThunderServe separates prefill and decode tasks onto different "model serving groups." This allows the system to match the workload to the hardware (e.g., using high-compute GPUs for prefill and high-bandwidth GPUs for decoding).
  • Two-Level Hierarchical Scheduling:
    • Upper Level: Uses a Tabu search algorithm to partition available GPUs into groups and designate them as either prefill or decode replicas.
  • Lower Level: Optimizes the internal parallel configuration (Tensor and Pipeline Parallelism) for each group and uses a Two-Stage Transportation Problem (TSTP) formulation to orchestrate request routing between prefill and decode replicas.
  • Lightweight Re-scheduling: To handle "workload shifts" or node failures, ThunderServe can swap the roles of existing replicas (e.g., turning a prefill group into a decode group) in seconds. This avoids the massive overhead of re-calculating a plan from scratch and reloading model weights, which can take minutes.
  • KV Cache Compression: To mitigate the slow interconnects in cloud environments, the system quantizes the Key-Value (KV) cache to 4-bit precision for transmission between nodes, dequantizing it only for computation to maintain model accuracy.

Key Insights and Ideas

  • Hardware-Phase Matching: The "Aha!" moment is that an older or "lesser" GPU might actually be better for a specific phase. For example, an NVIDIA 3090Ti has excellent memory bandwidth, making it a cost-effective choice for the decode phase, even if it lacks the raw TFLOPS of an A100.
Comparison of price efficiency between 3090 Ti and A40 GPUs for prefill and decode phases.
  • The Communication Tax: In the cloud, the network is the bottleneck. The authors found that without KV cache compression and smart orchestration, the cost of moving data between prefill and decode replicas can negate the benefits of phase splitting.
  • Role Flipping over Reloading: By simply changing the designation of a GPU group rather than its configuration, the system can adapt to dynamic traffic (like a sudden surge in long-form coding requests) with almost zero downtime.

Key Results

  • Efficiency: Given the same price budget, ThunderServe outperformed HexGen (a heterogeneous baseline) by 1.5× to 1.8× in throughput.
  • Cost-Effectiveness: Compared to industry standards like vLLM and DistServe running on homogeneous in-house clusters, ThunderServe achieved up to a 2.1× throughput increase and a 1.5× average reduction in latency deadlines.
  • Resilience: The lightweight re-scheduling mechanism finished in roughly 13 seconds, compared to over 150 seconds for a full system restart and parameter reload.
  • Accuracy: 4-bit KV cache compression reduced communication overhead by up to 30% while keeping the accuracy drop under 2% across all tested tasks.

Limitations and Future Work

The authors acknowledge a few areas for further exploration:

  • Static Budgeting: The current scheduler assumes a fixed pool of GPUs. Future iterations could integrate with cloud auto-scaling to dynamically rent or release GPUs based on demand.
  • Quantization Limits: While 4-bit quantization worked well, the system uses a "one-shot" approach. Exploring quantization-aware training or more sophisticated compression could potentially push this to 2-bit without significant quality loss.
  • State Management: The paper focuses on throughput and latency but doesn't deeply explore the complexities of state persistence or recovery for the KV cache if a node fails mid-generation.

Summary updated

Abstract

Recent developments in large language models (LLMs) have demonstrated their remarkable proficiency in a range of tasks. Compared to in-house homogeneous GPU clusters, deploying LLMs in cloud environments with diverse types of GPUs is crucial for addressing the GPU shortage problem and being more cost-effective. However, the diversity of network environments and various GPU types on the cloud bring difficulties to achieving high-performance serving. In this work, we propose ThunderServe, a high-performance and cost-efficient LLM serving system for heterogeneous cloud environments. We introduce a novel scheduling algorithm, which optimizes the deployment plan of LLM serving to accommodate the heterogeneous resource and network bandwidth conditions in cloud environments. Furthermore, we propose a lightweight re-scheduling mechanism, designed to adapt to fluctuating online conditions (e.g., node failures, workload shifts) without the need for costly restarts of ongoing services. Empirical results in both heterogeneous cloud and homogeneous in-house environments reveal that ThunderServe delivers up to a 2.1$\times$ and on average a $1.7\times$ increase in throughput and achieves up to a 2.5$\times$ and on average a $1.5\times$ reduction in latency deadlines compared with state-of-the-art systems given the same price budget, suggesting opting for cloud services provides a more cost-efficient solution.
BibTeX
@inproceedings{jiang2025thunderserve,
  author = {YOUHE JIANG and Fangcheng Fu and Xiaozhe Yao and Taiyi Wang and Bin CUI and Ana Klimovic and Eiko Yoneki},
  title = {{ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments}},
  booktitle = {MLSys},
  year = {2025},
  url = {https://openreview.net/forum?id=44PwmgOpAt},
}
Figure