Maximizing GPU Efficiency via Optimal Adapter Caching: An Analytical Approach for Multi-Tenant LLM Serving

Chen Wang, Ferran Agullo, Joan Oliveras, Chen Wang 0039, Alberto Gutierrez-Torre, Olivier Tardieu, Alaa Youssef, Jordi Torres, Josep Lluís BerralarXiv 2025 · DOI · arXiv

Abstract

Serving numerous specialized Large Language Model (LLM) adapters on a single GPU requires a delicate balance between maximizing throughput and avoiding memory exhaustion. This paper presents a high-fidelity Digital Twin (DT) that simulates LLM serving processes 90x faster than real-time. By leveraging this DT to train interpretable machine learning models, the authors achieve less than 7.2% error in predicting optimal serving configurations, effectively navigating the performance "cliff" associated with adapter caching.

Introduction: The Adapter Caching Problem

In multi-tenant LLM serving, a single base model (e.g., Llama-3) is served alongside hundreds of specialized Low-Rank Adaptation (LoRA) adapters. Although these adapters are small, they consume GPU memory for weights and KV caches.

System throughput in these environments follows a distinct "cliff" pattern. Initially, increasing the number of concurrent adapters improves throughput by aggregating requests. However, once memory saturation is reached, the system can no longer process requests within limits, leading to request starvation. At this point, the system spends more time swapping data than generating tokens, causing performance to collapse. Finding the "sweet spot"—the maximum number of concurrent adapters before this collapse—traditionally requires hours of expensive benchmarking on high-end hardware.

Throughput vs. number of adapters across varying parameters, illustrating the performance "cliff".

Methodology

The authors propose a data-driven approach to optimize serving configurations without the need for constant physical benchmarking:

  1. Digital Twin (DT): An offline emulator replicates the vLLM framework's inner workings, including its First-Come-First-Served scheduler, memory allocation, and Least Recently Used (LRU) adapter swapping logic. The DT runs on a single CPU core with minimal RAM, providing a 90x speedup over real-time execution.
  2. Synthetic Data Generation: Using the DT, the authors generated a dataset of over 87,000 runs, varying adapter ranks (8, 16, 32), arrival rates, and request lengths (using the ShareGPT dataset for realistic heterogeneity).
  3. Interpretable ML Models: Shallow decision trees and linear models were trained on this synthetic data to predict the optimal number of "adapter slots" (parallelism) and concurrent adapters. These models allow for human-readable decision rules, essential for production debugging.

Key Results and Insights

The study reveals that the saturation point is a dynamic interaction between adapter size, request length, and arrival rate. A system might handle 300 adapters with short prompts but fail with only 50 adapters if prompts are long.

Performance metrics for the Digital Twin and ML models across Llama and Qwen architectures.

As shown in the table above, the Digital Twin achieves high fidelity, matching real-system throughput within 5.1% error and Inter-Token Latency (ITL) within 9.6% error. Furthermore, the ML models predicted the optimal number of concurrent adapters with a maximum average error of only 1.0%. Inference for these models takes less than 0.15ms, enabling real-time re-optimization as traffic patterns shift.

Conclusion

This research demonstrates that high-fidelity simulation can replace expensive hardware benchmarking for optimizing LLM serving. While currently focused on single-GPU setups and the vLLM scheduler, future work aims to extend these analytical models to multi-GPU environments and broader serving frameworks.

Summary updated

BibTeX
@misc{wang2025maximizing,
  author = {Chen Wang and Ferran Agullo and Joan Oliveras and Chen Wang and Alberto Gutierrez-Torre and Olivier Tardieu and Alaa Youssef and Jordi Torres and Josep Lluís Berral},
  title = {{Maximizing GPU Efficiency via Optimal Adapter Caching: An Analytical Approach for Multi-Tenant LLM Serving}},
  year = {2025},
  eprint = {2508.08343},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.08343},
}
Figure