TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems

Feiyang Wu, Zhuohang Bian, Guoyang Duan, Tianle Xu, Junchi Wu, Teng Ma, Yongqiang Yao, Ruihao Gong, Youwei Zhuo

Abstract

Serving Large Language Models (LLMs) at scale presents significant resource challenges, necessitating sophisticated software optimizations and specialized hardware. TokenSim is introduced as a modular, event-driven simulator designed to explore the complex interplay between hardware and software in LLM inference systems. Built on the SimPy library, TokenSim achieves high fidelity with an error rate of less than 1% compared to real-world hardware while supporting advanced techniques such as continuous batching, PagedAttention, and disaggregated architectures.

Introduction

The demand for LLM serving has led to the development of complex system-level optimizations, including dynamic request scheduling and specialized memory management. Existing simulators often lack the granularity to model these dynamic, multi-user environments accurately or the flexibility to support experimental optimizations. TokenSim addresses this gap by providing a tool that can accurately simulate thousands of concurrent requests with varying lengths, helping researchers guide the design of next-generation inference stacks.

System Architecture

TokenSim's architecture is highly modular, allowing for detailed simulation of various system components without the need for high-end GPU resources during the simulation phase itself.

TokenSim system overview showing the interaction between workload, schedulers, and hardware workers.

Key components of the system include:

  • Two-Stage Scheduler: A Global Scheduler dispatches requests to specialized workers (e.g., prefill vs. decode), while Local Schedulers manage execution queues using continuous batching.
  • Fine-Grained Memory Manager: Essential for simulating PagedAttention, this component tracks memory utilization at the block, token, or byte level.
  • Operator-Level Breakpoints: Users can insert hooks into model configurations to trigger custom logic, such as data movement in disaggregated setups.
  • Extensible Compute Models: TokenSim integrates with existing hardware simulators like GenZ or Roofline models to estimate operation execution times.

Methodology and Validation

Validation was performed using the ShareGPT dataset, utilizing up to 50,000 requests to test realistic distributions of prompt and response lengths. TokenSim was benchmarked against vLLM running on NVIDIA A100 GPUs. The results demonstrated exceptional accuracy, with a geometric mean error of only 0.109% for throughput and less than 0.6% for latency percentiles (P50, P99).

Validation results showing TokenSim (T-) closely matching vLLM (V-) in both throughput and latency across different Queries Per Second (QPS).

Key Insights

Simulation experiments using TokenSim revealed several critical design considerations for LLM systems:

  • Throughput vs. Tail Latency: Strategically limiting new requests to reserve memory for existing ones can significantly reduce request preemptions and improve tail latency, despite a potential decrease in raw throughput.
  • Hardware Specialization: The "Prefill" stage is compute-bound and benefits from high FLOPS, whereas the "Decode" stage is memory-bandwidth bound. The optimal ratio of prefill-to-decode hardware depends heavily on the expected output length of requests.
  • Alternative Hardware: Processing-In-Memory (PIM) chips were identified as a cost-effective alternative for the decode stage, provided physical slot limitations on server boards are addressed.

Conclusion

TokenSim provides a highly accurate and efficient platform for exploring the LLM inference design space. While some modeling challenges remain regarding hardware bus communication fluctuations, the simulator's modularity and speed make it a valuable tool for future hardware-software co-design in heterogeneous clusters.

Summary updated

BibTeX
@inproceedings{wu2025tokensim,
  author = {Feiyang Wu and Zhuohang Bian and Guoyang Duan and Tianle Xu and Junchi Wu and Teng Ma and Yongqiang Yao and Ruihao Gong and Youwei Zhuo},
  title = {{TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems}},
  booktitle = {APPT},
  year = {2025},
  doi = {10.1007/978-981-95-1021-4\_19},
}
Figure