Abstract
LLMs are increasingly used inside database systems and in database applications for better complexity management and decision-making, where LLM inferences require significant GPU costs. LLM inference systems, however, are slow compared to database systems, limiting the expansion of the use of LLMs inside database systems. This paper first analyzes the LLM inference performance and focuses on a data management issue in LLM inference. We reveal that the root of the problem is the lack of an adequate resource cost model and optimization strategy when executing multiple concurrent inference requests. We adapt classic database multi-query optimization techniques by introducing cost models for concurrent inference requests and new scheduling strategies to optimize the use of memory resources by concurrent requests, thereby substantially improving performance.
Introduction
Large Language Model (LLM) inference has become a significant computational burden. Unlike training, which is a massive one-time expense, inference is an ongoing cost that scales with usage. As models grow in complexity, the efficiency of serving these models becomes paramount. The primary bottleneck is often not raw compute power, but the management of the KV Cache (Key-Value cache) within finite GPU memory. When concurrent requests exceed memory capacity, systems must preempt requests, leading to wasted computation and increased latency.
Methodology: The InferMax Framework
To address these challenges, we introduce InferMax, a framework designed to bridge the gap between database resource management and LLM serving. InferMax integrates cost modeling, advanced scheduling, and preemption policies to optimize GPU resource utilization.
The framework relies on four key pillars:
- Cost Modeling: Linear models predict batch execution times based on FLOPs and data movement, proving that simple models are often sufficient for complex scheduling.
- Shortest-Request First (SRF) Policy: A preemption strategy that prioritizes requests with the lowest "refill cost"—the time required to recompute their KV cache.
- The Five-Minute Rule: An adaptation of Jim Gray’s database principle to determine the economic break-even point for caching versus recomputation.
- Constraint Satisfaction Problem (CSP): A mathematical formulation used to find theoretically optimal schedules for benchmarking practical policies.
Core Concepts: Memory and Preemption
The fundamental challenge in LLM inference is managing the GPU memory wall. Memory bandwidth often limits performance more than capacity. When memory is full, the system must decide which request to preempt to make room for others.
Our research suggests that preemption should be viewed as a tool rather than a failure. By preempting "short" requests under high contention, the system can minimize the total recomputation overhead, as short requests are cheaper to restart than long ones. This "Refill Paradox" highlights that long requests, which have quadratically growing recomputation costs, should be prioritized for completion once they have progressed significantly.
Experimental Results
We evaluated our approach using real-world traces (AzureConv) and long-form generation datasets across various models (Llama-2-7B to Llama-3-70B) on A100 and H100 GPUs.
- Latency Reduction: The SRF policy reduced inference latency by up to 15% in real-world deployments and up to 40% in high-contention simulations.
- Economic Impact: For large-scale services, these optimizations could save over $3 million in monthly GPU costs for providers at the scale of ChatGPT.
- Model Accuracy: Our linear cost models achieved high predictive accuracy, typically staying within 2% of actual GPU runtimes.
Conclusion and Future Work
By treating the GPU as a database buffer and applying proven data management strategies, we can significantly improve LLM inference efficiency. Future work will explore the complexities of "prefix sharing" (KV reuse across different requests) and the potential for "KV swapping" to CPU memory or SSDs as hardware interconnects like PCIe and NVLink continue to evolve, further reducing the cost of preemption.
Summary updated