Abstract
Serving numerous specialized Large Language Model (LLM) adapters on a single GPU requires a delicate balance between maximizing throughput and avoiding memory exhaustion. This paper presents a high-fidelity Digital Twin (DT) that simulates LLM serving processes 90x faster than real-time. By leveraging this DT to train interpretable machine learning models, the authors achieve less than 7.2% error in predicting optimal serving configurations, effectively navigating the performance "cliff" associated with adapter caching.
Introduction: The Adapter Caching Problem
In multi-tenant LLM serving, a single base model (e.g., Llama-3) is served alongside hundreds of specialized Low-Rank Adaptation (LoRA) adapters. Although these adapters are small, they consume GPU memory for weights and KV caches.
System throughput in these environments follows a distinct "cliff" pattern. Initially, increasing the number of concurrent adapters improves throughput by aggregating requests. However, once memory saturation is reached, the system can no longer process requests within limits, leading to request starvation. At this point, the system spends more time swapping data than generating tokens, causing performance to collapse. Finding the "sweet spot"—the maximum number of concurrent adapters before this collapse—traditionally requires hours of expensive benchmarking on high-end hardware.
Methodology
The authors propose a data-driven approach to optimize serving configurations without the need for constant physical benchmarking:
- Digital Twin (DT): An offline emulator replicates the vLLM framework's inner workings, including its First-Come-First-Served scheduler, memory allocation, and Least Recently Used (LRU) adapter swapping logic. The DT runs on a single CPU core with minimal RAM, providing a 90x speedup over real-time execution.
- Synthetic Data Generation: Using the DT, the authors generated a dataset of over 87,000 runs, varying adapter ranks (8, 16, 32), arrival rates, and request lengths (using the ShareGPT dataset for realistic heterogeneity).
- Interpretable ML Models: Shallow decision trees and linear models were trained on this synthetic data to predict the optimal number of "adapter slots" (parallelism) and concurrent adapters. These models allow for human-readable decision rules, essential for production debugging.
Key Results and Insights
The study reveals that the saturation point is a dynamic interaction between adapter size, request length, and arrival rate. A system might handle 300 adapters with short prompts but fail with only 50 adapters if prompts are long.
As shown in the table above, the Digital Twin achieves high fidelity, matching real-system throughput within 5.1% error and Inter-Token Latency (ITL) within 9.6% error. Furthermore, the ML models predicted the optimal number of concurrent adapters with a maximum average error of only 1.0%. Inference for these models takes less than 0.15ms, enabling real-time re-optimization as traffic patterns shift.
Conclusion
This research demonstrates that high-fidelity simulation can replace expensive hardware benchmarking for optimizing LLM serving. While currently focused on single-GPU setups and the vLLM scheduler, future work aims to extend these analytical models to multi-GPU environments and broader serving frameworks.
Summary updated