Introduction
Modern Large Language Models (LLMs) have grown to sizes that far exceed the High Bandwidth Memory (HBM) capacity of even the most advanced GPUs. For instance, models like LLaMA-70B or Qwen2-72B require significantly more than the 80GB available on an NVIDIA A100, creating a massive memory gap for inference.
To bridge this gap, "memory offloading" techniques store model weights in CPU memory and transfer them to the GPU on demand. However, existing solutions often struggle to maintain the strict latency Service-Level Objectives (SLOs) required for interactive applications like chatbots. Systems like DeepSpeed Inference offload aggressively, leading to latency spikes, while others like FlexGen are overly conservative, sacrificing throughput.
The Select-N Approach
Select-N introduces a dynamic memory offloading system designed to maximize throughput while strictly adhering to latency SLOs. The core innovation is the offloading interval. Instead of a static offloading strategy, Select-N groups model layers into intervals of $i$ layers. For each interval, only the final layer's state is offloaded. By initiating the prefetch of this offloaded layer as soon as computation begins on the first layer of the interval, Select-N effectively hides PCIe transfer latency behind GPU computation.
The system operates in two stages:
- Offline Performance Analyzer: Profiles the model to map input sizes and SLOs to optimal offloading intervals. Because LLM layers have identical structures, their computation time is highly deterministic, making offline profiling extremely accurate.
- Online Bandwidth Coordinator: Monitors real-time PCIe bandwidth contention and adjusts offloading intervals dynamically to ensure no SLOs are violated while maximizing total memory offloaded.
Key Insights
The effectiveness of Select-N rests on several critical observations:
- Deterministic Computation: LLM inference latency is highly predictable because Transformer layers perform uniform operations. This allows for accurate offline profiling.
- Deep Prefetching: Prefetching across multi-layer intervals provides a larger window to mask slow data transfers compared to layer-by-layer approaches.
- Phase Decoupling: Prefill and decoding phases have distinct bottlenecks (compute-bound vs. memory-bound). Select-N optimizes them independently with tailored offloading intervals.
Evaluation and Results
The authors evaluated Select-N against state-of-the-art systems using models like OPT and LLaMA. Select-N demonstrated superior performance in both SLO compliance and efficiency.
Key results include:
- Throughput Gains: Select-N achieved up to a 1.85x throughput increase over FlexGen by enabling larger batch sizes through more efficient memory utilization. As shown in the throughput evaluation, Select-N scales significantly better with batch size compared to aggressive offloading methods like DeepSpeed.
- SLO Compliance: While competitors like DeepSpeed frequently violated latency targets (by up to 8.08x), Select-N consistently met both Time to First Token (TTFT) and Time Per Output Token (TPOT) requirements.
- Memory Efficiency: By avoiding overly conservative estimations, Select-N offloaded 2.37x more memory than static methods like FlexGen.
Conclusion
Select-N represents a significant step forward in making large-scale LLM inference more accessible and cost-effective. By leveraging the deterministic nature of LLM workloads and implementing dynamic, interval-based offloading, it successfully balances the competing demands of high throughput and low latency. Future work may extend these techniques to heterogeneous architectures like Mixture-of-Experts or distributed multi-node environments.
Summary updated