TL;DR
Serving multiple specialized Large Language Models (LLMs) efficiently is a growing challenge. This paper introduces TOPPINGS, a system that uses CPUs to assist GPUs in serving these models, significantly reducing latency and improving the ability to meet performance guarantees. It also incorporates a smart scheduling system that considers the "rank" of different model adaptations to optimize resource usage across multiple servers.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly being adapted for specific tasks using techniques like Low-Rank Adaptation (LoRA). This creates many specialized "adapters" that need to be served efficiently, especially in multi-tenant environments where many users might request different adaptations simultaneously. The core problem is that loading these adapters onto GPUs is slow, causing delays (time-to-first-token) and interrupting ongoing model generations. Existing systems struggle with this "cold-start" problem and also face challenges in efficiently scheduling requests for adapters with varying complexities (ranks) across multiple servers.
Methods
TOPPINGS tackles these challenges with two main innovations:
- CPU-Assisted LoRA Serving: Instead of waiting for a LoRA adapter to fully load onto the GPU, TOPPINGS uses the CPU to start computing the adapter's adaptation in parallel. This "CPU-assisted" approach overlaps the adapter loading time with computation, hiding the latency. This is made possible by efficient coordination between CPU and GPU processes, using techniques like asynchronous memory copies and shared memory for fast data exchange.
- Rank-Aware Request Scheduling: The system employs a scheduler that understands how different LoRA adapter ranks affect performance. It uses a performance model to predict the latency of serving requests with varying adapter ranks and then intelligently routes incoming requests to the best-suited inference server to meet Service Level Objectives (SLOs).
Datasets
The paper uses the following datasets for evaluation:
- MAF trace [40]: A scaled production workload emulating real-world model serving.
- LMSYS-CHAT-1M [61]: A dataset of real-world conversations for prompt and output lengths.
- QMSum [46, 62]: A benchmark for query-based multi-domain meeting summarization, used for testing with long prompts.
Key Insights and Ideas
- CPUs are not just for background tasks: The paper demonstrates that CPUs, often underutilized in LLM serving, can be effectively leveraged to accelerate critical parts of the inference pipeline, specifically LoRA adaptation.
- Overlapping is key: By overlapping the slow adapter loading process with computation on the CPU, TOPPINGS effectively hides latency that would otherwise impact user experience.
- Rank matters for scheduling: The complexity (rank) of LoRA adapters significantly impacts serving performance. A scheduler that accounts for this heterogeneity can achieve much better overall system performance and SLO attainment.
- Efficiency gains are substantial: The combined approach of CPU-assisted serving and rank-aware scheduling leads to significant improvements in latency and SLO attainment compared to existing methods.
Key Results
- TOPPINGS improves average request serving latency by up to 1.7x compared to state-of-the-art LoRA serving systems.
- Achieves an SLO attainment of up to 99%.
- Reduces the cumulative decoding-interruption overhead, which can account for up to 29% of the total request serving time in some scenarios.
- In microbenchmarks, the sync-free CPU LoRA invocation kernel reduces prefill latency by up to 15%.
- The rank-aware scheduler improves SLO attainment by 21% and 26% at higher request rates compared to rank-agnostic schedulers.
- When serving 512 adapters, dLoRA causes GPU Out-of-Memory (OOM) errors, while TOPPINGS remains stable.
Limitations and Future Work
- Performance Model Retraining: The performance models used for scheduling are hardware and model-specific. They require re-profiling if the hardware or LLM changes.
- Fairness: While TOPPINGS is compatible with fairness strategies, it doesn't explicitly address fairness concerns like resource allocation among users. The authors note that fairness is a broader challenge in LLM serving.
- Future Work: The paper suggests that the CPU-assisted design is versatile and can be adapted to other workloads. Further exploration of its compatibility with different LLMs and frameworks is implied. The authors also mention that redundant CPU processes can enhance fault tolerance, hinting at future work in that area.