lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models

Haoxin Wang, Haoxin Wang 0003, Xiaolong Tu, Hongyu Ke, Huirong Chai, Dawei Chen, Kyungtae HanarXiv 2025 · DOI · arXiv

Abstract

Deploying Large Language Models (LLMs) on mobile devices offers significant advantages for user privacy and operational costs, yet performance optimization remains hindered by a lack of precise, low-overhead profiling tools. We present lm-Meter, a real-time latency profiler designed specifically for on-device LLMs. By instrumenting the inference pipeline at both the phase and GPU kernel levels, lm-Meter achieves high accuracy with negligible overhead. Our analysis reveals that, unlike server-grade deployments, the "prefill" phase is the primary performance bottleneck on mobile hardware, and mobile GPUs remain idle for over 20% of the inference cycle due to scheduling inefficiencies.

1. Introduction

The transition of Large Language Models (LLMs) from massive data centers to edge devices like smartphones represents a critical shift toward privacy-preserving, offline-capable AI. However, mobile environments are constrained by strict memory, compute, and thermal limits. To optimize models for these environments, engineers require granular visibility into execution latency.

Existing mobile profiling tools, such as Android GPU Inspector or Perfetto, are often ill-suited for this task. They typically require tethered host machines for offline analysis or introduce significant system overhead that distorts the very performance they aim to measure. Without a lightweight, online profiler capable of distinguishing between specific LLM phases—such as embedding, prefill, and decoding—researchers are effectively "flying blind" when optimizing on-device models.

2. The lm-Meter Framework

To address these challenges, we developed lm-Meter, integrated within the MLC LLM framework. It provides visibility at two distinct granularities:

  • Phase-Level Profiling: By instrumenting the native C++ backend with lightweight monotonic clocks (std::chrono::steady_clock), lm-Meter captures the duration of high-level semantic phases (prefill, decode, sampling). This approach avoids the latency jitter and inaccuracy associated with high-level application wrappers (e.g., Java or Kotlin).
  • Kernel-Level Profiling: To gain insight into GPU execution without relying on closed-source drivers, lm-Meter hooks into the event callback mechanisms of standard libraries like OpenCL, Vulkan, and Metal. This allows for the capture of precise timestamps for when a kernel is queued, submitted, and executed on the GPU.

To validate the system's accuracy, we employed a "kernel duplication" technique. By forcing the compiler to execute target kernels 50 to 1,000 times in succession, we created a massive, measurable ground-truth baseline to compare against lm-Meter’s real-time readings.

3. Benchmarking Methodology

We evaluated lm-Meter across a diverse suite of compact LLMs, including Pythia, SmolLM2, Qwen1.5, Llama-3.2, Gemma-2, TinyLlama, and DeepSeek-R1-Distill. Performance was measured using standard benchmarks such as ARC (Easy/Challenge), HellaSwag, and GSM8K.

A significant challenge in mobile benchmarking is thermal throttling; running thousands of prompts can cause the device to overheat, ruining the data. To mitigate this, we implemented a Kullback-Leibler (KL) divergence-based sampling algorithm. This allowed us to extract a small, representative subset (1% to 10%) of the datasets that maintained the original token-length distribution, ensuring stable and accurate on-device testing without triggering thermal limits.

4. Key Insights: The Inverted Bottleneck

Our empirical study using lm-Meter yielded several surprising insights into the nature of on-device inference:

4.1 Prefill vs. Decode

In server-side deployments (e.g., using NVIDIA A100s), the autoregressive decoding phase is typically the primary bottleneck due to memory bandwidth constraints. On mobile devices, we observed the opposite: prefill is the dominant bottleneck. Processing the initial input context in a single pass places immense pressure on mobile compute units. As models scale from 70M to 1.4B parameters, decode latency increases by 10x, whereas prefill latency suffers a staggering 158x slowdown.

4.2 Architecture and Quantization

We found that model architecture often outweighs raw parameter count on the edge. For instance, the optimized SmolLM2-360M frequently outperformed the larger Pythia-1B. Additionally, we introduced the Harmonic Quantization (HQ) score to evaluate the trade-off between accuracy and latency in 4-bit quantization. Our results indicate that while quantization is highly effective for complex reasoning tasks (GSM8K), it provides diminishing returns for simpler tasks like ARC-Easy.

4.3 GPU Underutilization

Kernel-level profiling revealed that mobile GPUs are idle for approximately 21% of the time during the decoding phase. This inefficiency is largely attributed to host-side data preparation and I/O stalls, suggesting that improved CPU-GPU scheduling could yield significant performance gains.

5. Performance and Accuracy

lm-Meter demonstrates exceptional performance characteristics:

  • Accuracy: It achieves ≥ 99.99% accuracy for phase-level profiling and 95-99% for kernel-level profiling compared to heavy offline tools.
  • Low Overhead: Even under "Powersave" CPU configurations, the tool reduces prefill throughput by only 2.58% and decode throughput by 0.99%. In contrast, existing state-of-the-art profilers can reduce decode throughput by over 93%.

6. Conclusion and Future Work

lm-Meter provides the granular, real-time data necessary to optimize LLMs for the edge. While our current study focused on Google Pixel devices and 4-bit quantization, the framework is designed for extensibility. Future work will involve expanding the hardware scope to include NPUs and other edge platforms, integrating energy consumption profiling, and developing an "Adaptation Engine" that uses real-time telemetry to dynamically adjust model execution parameters on the fly.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly integrated into everyday applications, but their prevalent cloud-based deployment raises growing concerns around data privacy and long-term sustainability. Running LLMs locally on mobile and edge devices (on-device LLMs) offers the promise of enhanced privacy, reliability, and reduced communication costs. However, realizing this vision remains challenging due to substantial memory and compute demands, as well as limited visibility into performance-efficiency trade-offs on resource-constrained hardware. We propose lm-Meter, the first lightweight, online latency profiler tailored for on-device LLM inference. lm-Meter captures fine-grained, real-time latency at both phase (e.g., embedding, prefill, decode, softmax, sampling) and kernel levels without auxiliary devices. We implement lm-Meter on commercial mobile platforms and demonstrate its high profiling accuracy with minimal system overhead, e.g., only 2.58% throughput reduction in prefill and 0.99% in decode under the most constrained Powersave governor. Leveraging lm-Meter, we conduct comprehensive empirical studies revealing phase- and kernel-level bottlenecks in on-device LLM inference, quantifying accuracy-efficiency trade-offs, and identifying systematic optimization opportunities. lm-Meter provides unprecedented visibility into the runtime behavior of LLMs on constrained platforms, laying the foundation for informed optimization and accelerating the democratization of on-device LLM systems. Code and tutorials are available at https://github.com/amai-gsu/LM-Meter.
BibTeX
@misc{wang2025lm,
  author = {Haoxin Wang and Haoxin Wang and Xiaolong Tu and Hongyu Ke and Huirong Chai and Dawei Chen and Kyungtae Han},
  title = {{lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models}},
  year = {2025},
  eprint = {2510.06126},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.06126},
}

Loading related content...

Figure