Memory Offloading for Large Language Model Inference with Latency SLO Guarantees

Chenxiang Ma, Zhisheng Ye, Hanyu Zhao, Zehua Yang, Tianhao Fu, Jiaxun Han, Jie Zhang, Yingwei Luo, Xiaolin Wang, Zhenlin Wang, Yong Li, Diyu ZhouarXiv 2025 · DOI · arXiv

Introduction

Modern Large Language Models (LLMs) have grown to sizes that far exceed the High Bandwidth Memory (HBM) capacity of even the most advanced GPUs. For instance, models like LLaMA-70B or Qwen2-72B require significantly more than the 80GB available on an NVIDIA A100, creating a massive memory gap for inference.

Memory usage of popular LLMs compared to the 80GB HBM capacity of high-end GPUs.

To bridge this gap, "memory offloading" techniques store model weights in CPU memory and transfer them to the GPU on demand. However, existing solutions often struggle to maintain the strict latency Service-Level Objectives (SLOs) required for interactive applications like chatbots. Systems like DeepSpeed Inference offload aggressively, leading to latency spikes, while others like FlexGen are overly conservative, sacrificing throughput.

The Select-N Approach

Select-N introduces a dynamic memory offloading system designed to maximize throughput while strictly adhering to latency SLOs. The core innovation is the offloading interval. Instead of a static offloading strategy, Select-N groups model layers into intervals of $i$ layers. For each interval, only the final layer's state is offloaded. By initiating the prefetch of this offloaded layer as soon as computation begins on the first layer of the interval, Select-N effectively hides PCIe transfer latency behind GPU computation.

The system operates in two stages:

  1. Offline Performance Analyzer: Profiles the model to map input sizes and SLOs to optimal offloading intervals. Because LLM layers have identical structures, their computation time is highly deterministic, making offline profiling extremely accurate.
  2. Online Bandwidth Coordinator: Monitors real-time PCIe bandwidth contention and adjusts offloading intervals dynamically to ensure no SLOs are violated while maximizing total memory offloaded.

Key Insights

The effectiveness of Select-N rests on several critical observations:

  • Deterministic Computation: LLM inference latency is highly predictable because Transformer layers perform uniform operations. This allows for accurate offline profiling.
  • Deep Prefetching: Prefetching across multi-layer intervals provides a larger window to mask slow data transfers compared to layer-by-layer approaches.
  • Phase Decoupling: Prefill and decoding phases have distinct bottlenecks (compute-bound vs. memory-bound). Select-N optimizes them independently with tailored offloading intervals.

Evaluation and Results

The authors evaluated Select-N against state-of-the-art systems using models like OPT and LLaMA. Select-N demonstrated superior performance in both SLO compliance and efficiency.

Throughput comparison between SELECT-N and DeepSpeed across varying batch sizes.

Key results include:

  • Throughput Gains: Select-N achieved up to a 1.85x throughput increase over FlexGen by enabling larger batch sizes through more efficient memory utilization. As shown in the throughput evaluation, Select-N scales significantly better with batch size compared to aggressive offloading methods like DeepSpeed.
  • SLO Compliance: While competitors like DeepSpeed frequently violated latency targets (by up to 8.08x), Select-N consistently met both Time to First Token (TTFT) and Time Per Output Token (TPOT) requirements.
  • Memory Efficiency: By avoiding overly conservative estimations, Select-N offloaded 2.37x more memory than static methods like FlexGen.

Conclusion

Select-N represents a significant step forward in making large-scale LLM inference more accessible and cost-effective. By leveraging the deterministic nature of LLM workloads and implementing dynamic, interval-based offloading, it successfully balances the competing demands of high throughput and low latency. Future work may extend these techniques to heterogeneous architectures like Mixture-of-Experts or distributed multi-node environments.

Summary updated

BibTeX
@misc{ma2025memory,
  author = {Chenxiang Ma and Zhisheng Ye and Hanyu Zhao and Zehua Yang and Tianhao Fu and Jiaxun Han and Jie Zhang and Yingwei Luo and Xiaolin Wang and Zhenlin Wang and Yong Li and Diyu Zhou},
  title = {{Memory Offloading for Large Language Model Inference with Latency SLO Guarantees}},
  year = {2025},
  eprint = {2502.08182},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.08182},
}
Figure