ODMA: On-Demand Memory Allocation Framework for LLM Serving on LPDDR-Class Accelerators

Guoqiang Zou, Wanyu Wang, Hao Zheng, Longxiang Yin, Yinhe HanarXiv 2025 · DOI · arXiv

Introduction

Serving Large Language Models (LLMs) requires efficient management of the Key-Value (KV) cache, which stores intermediate states for active requests. While high-end GPUs utilize High Bandwidth Memory (HBM) and paging techniques like PagedAttention, many cost-effective accelerators rely on LPDDR5 memory. These LPDDR-based systems, characterized as Random-Access-Constrained Device Memory (RACM), suffer significant performance degradation with the small, random memory accesses required by paging.

Consequently, these systems often resort to "static pre-allocation," reserving memory for the maximum possible sequence length for every request. This leads to massive memory waste, low concurrency, and poor throughput. ODMA (On-Demand Memory Allocation) addresses this by introducing a hardware-aware, dynamic allocation framework.

System Architecture

ODMA replaces static allocation with a three-component pipeline designed for RACM hardware:

  1. Generation-Length Predictor: A lightweight model (DistilBERT-scale) analyzes incoming prompts to estimate the number of tokens to be generated and provides an uncertainty score.
  2. Dynamic Bucket Manager: Instead of fixed sizes, ODMA maintains an online histogram of request lengths and periodically recalculates bucket boundaries (e.g., small, medium, large) based on real-time traffic distributions.
  3. Large-Bucket Safety Strategy: To handle unpredictable or long-tail requests, ODMA uses a "large bucket" fallback. If a request exceeds its predicted bucket, it is moved via a contiguous "allocate-and-copy" operation, which remains efficient on LPDDR.
ODMA System Architecture

Evaluation

The framework was evaluated using the DeepSeek-R1-Distill-Qwen-7B model on Cambricon MLU370-X4 accelerators across two datasets:

  • Alpaca: Instruction-following tasks.
  • Google-NQ (Natural Questions): Real-world search queries with a heavy-tailed length distribution.

Key Results

ODMA demonstrates superior performance over existing methods like the S3 predictor and standard static allocation baselines.

  • Prediction Accuracy: On the challenging Google-NQ dataset, ODMA improved length prediction accuracy from 82.68% (S3) to 93.36%.
  • Memory Utilization: Device-memory utilization increased from 55.05% to 72.45% on Alpaca and from 42.54% to 61.79% on Google-NQ.
  • Throughput: The framework delivered a 23% throughput increase on Alpaca and a 27% increase on Google-NQ compared to the Cambricon-vLLM static baseline.
Performance comparison: Accuracy and Throughput

Conclusion

ODMA proves that hardware-aware, contiguous memory allocation is essential for LLM serving on LPDDR-class accelerators. By combining predictive modeling with dynamic bucket management, the framework significantly reduces internal fragmentation and boosts system throughput without the overhead of random-access paging. Future work will focus on reducing predictor latency and scaling to distributed clusters.

Summary updated

BibTeX
@misc{zou2025odma,
  author = {Guoqiang Zou and Wanyu Wang and Hao Zheng and Longxiang Yin and Yinhe Han},
  title = {{ODMA: On-Demand Memory Allocation Framework for LLM Serving on LPDDR-Class Accelerators}},
  year = {2025},
  eprint = {2512.09427},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.09427},
}
Figure