LLM-on-the-Palm: Mobile LLM Inference with PIM-Enhanced NAND Flash Memory

Hyunjin Kim, Sanghyeok Han, Jae-Joon KimICCAD 2025 · OpenReview

Abstract

Large Language Model (LLM) inference on edge devices is crucial for democratizing AI and addressing privacy and security concerns associated with cloud services. However, the large parameter sizes of LLMs pose significant challenges in deployment on resource-constrained devices, affecting user experience. Mobile DRAM typically cannot accommodate these parameters, necessitating the use of NAND flash memory, but it has considerably slower access speeds. To overcome the limitations, we propose LLM-on-the-Palm, an LLM inference system for smartphones that leverages Processing-in-Memory (PIM)-enhanced NAND flash memory to enable fast inference of billion-parameter LLMs with minimal hardware modifications. Our approach requires only a single multiply-accumulate (MAC) unit per plane in a NAND flash memory - 256 MAC units for 256 GB SSD - without compromising cell density. Simulation results show that our design achieves ∼100 ms per token generation on a 6.7B model under specifications similar to an iPhone-15.
BibTeX
@inproceedings{kim2025llm,
  author = {Hyunjin Kim and Sanghyeok Han and Jae-Joon Kim},
  title = {{LLM-on-the-Palm: Mobile LLM Inference with PIM-Enhanced NAND Flash Memory}},
  booktitle = {ICCAD},
  year = {2025},
  url = {https://openreview.net/forum?id=D1Zee6m7h4},
}

Loading related content...

Figure