Introduction
Serving Large Language Models (LLMs) requires efficient management of the Key-Value (KV) cache, which stores intermediate states for active requests. While high-end GPUs utilize High Bandwidth Memory (HBM) and paging techniques like PagedAttention, many cost-effective accelerators rely on LPDDR5 memory. These LPDDR-based systems, characterized as Random-Access-Constrained Device Memory (RACM), suffer significant performance degradation with the small, random memory accesses required by paging.
Consequently, these systems often resort to "static pre-allocation," reserving memory for the maximum possible sequence length for every request. This leads to massive memory waste, low concurrency, and poor throughput. ODMA (On-Demand Memory Allocation) addresses this by introducing a hardware-aware, dynamic allocation framework.
System Architecture
ODMA replaces static allocation with a three-component pipeline designed for RACM hardware:
- Generation-Length Predictor: A lightweight model (DistilBERT-scale) analyzes incoming prompts to estimate the number of tokens to be generated and provides an uncertainty score.
- Dynamic Bucket Manager: Instead of fixed sizes, ODMA maintains an online histogram of request lengths and periodically recalculates bucket boundaries (e.g., small, medium, large) based on real-time traffic distributions.
- Large-Bucket Safety Strategy: To handle unpredictable or long-tail requests, ODMA uses a "large bucket" fallback. If a request exceeds its predicted bucket, it is moved via a contiguous "allocate-and-copy" operation, which remains efficient on LPDDR.
Evaluation
The framework was evaluated using the DeepSeek-R1-Distill-Qwen-7B model on Cambricon MLU370-X4 accelerators across two datasets:
- Alpaca: Instruction-following tasks.
- Google-NQ (Natural Questions): Real-world search queries with a heavy-tailed length distribution.
Key Results
ODMA demonstrates superior performance over existing methods like the S3 predictor and standard static allocation baselines.
- Prediction Accuracy: On the challenging Google-NQ dataset, ODMA improved length prediction accuracy from 82.68% (S3) to 93.36%.
- Memory Utilization: Device-memory utilization increased from 55.05% to 72.45% on Alpaca and from 42.54% to 61.79% on Google-NQ.
- Throughput: The framework delivered a 23% throughput increase on Alpaca and a 27% increase on Google-NQ compared to the Cambricon-vLLM static baseline.
Conclusion
ODMA proves that hardware-aware, contiguous memory allocation is essential for LLM serving on LPDDR-class accelerators. By combining predictive modeling with dynamic bucket management, the framework significantly reduces internal fragmentation and boosts system throughput without the overhead of random-access paging. Future work will focus on reducing predictor latency and scaling to distributed clusters.
Summary updated