Abstract
Deploying Large Language Models (LLMs) on mobile devices offers significant advantages for user privacy and operational costs, yet performance optimization remains hindered by a lack of precise, low-overhead profiling tools. We present lm-Meter, a real-time latency profiler designed specifically for on-device LLMs. By instrumenting the inference pipeline at both the phase and GPU kernel levels, lm-Meter achieves high accuracy with negligible overhead. Our analysis reveals that, unlike server-grade deployments, the "prefill" phase is the primary performance bottleneck on mobile hardware, and mobile GPUs remain idle for over 20% of the inference cycle due to scheduling inefficiencies.
1. Introduction
The transition of Large Language Models (LLMs) from massive data centers to edge devices like smartphones represents a critical shift toward privacy-preserving, offline-capable AI. However, mobile environments are constrained by strict memory, compute, and thermal limits. To optimize models for these environments, engineers require granular visibility into execution latency.
Existing mobile profiling tools, such as Android GPU Inspector or Perfetto, are often ill-suited for this task. They typically require tethered host machines for offline analysis or introduce significant system overhead that distorts the very performance they aim to measure. Without a lightweight, online profiler capable of distinguishing between specific LLM phases—such as embedding, prefill, and decoding—researchers are effectively "flying blind" when optimizing on-device models.
2. The lm-Meter Framework
To address these challenges, we developed lm-Meter, integrated within the MLC LLM framework. It provides visibility at two distinct granularities:
- Phase-Level Profiling: By instrumenting the native C++ backend with lightweight monotonic clocks (
std::chrono::steady_clock), lm-Meter captures the duration of high-level semantic phases (prefill, decode, sampling). This approach avoids the latency jitter and inaccuracy associated with high-level application wrappers (e.g., Java or Kotlin). - Kernel-Level Profiling: To gain insight into GPU execution without relying on closed-source drivers, lm-Meter hooks into the event callback mechanisms of standard libraries like OpenCL, Vulkan, and Metal. This allows for the capture of precise timestamps for when a kernel is queued, submitted, and executed on the GPU.
To validate the system's accuracy, we employed a "kernel duplication" technique. By forcing the compiler to execute target kernels 50 to 1,000 times in succession, we created a massive, measurable ground-truth baseline to compare against lm-Meter’s real-time readings.
3. Benchmarking Methodology
We evaluated lm-Meter across a diverse suite of compact LLMs, including Pythia, SmolLM2, Qwen1.5, Llama-3.2, Gemma-2, TinyLlama, and DeepSeek-R1-Distill. Performance was measured using standard benchmarks such as ARC (Easy/Challenge), HellaSwag, and GSM8K.
A significant challenge in mobile benchmarking is thermal throttling; running thousands of prompts can cause the device to overheat, ruining the data. To mitigate this, we implemented a Kullback-Leibler (KL) divergence-based sampling algorithm. This allowed us to extract a small, representative subset (1% to 10%) of the datasets that maintained the original token-length distribution, ensuring stable and accurate on-device testing without triggering thermal limits.
4. Key Insights: The Inverted Bottleneck
Our empirical study using lm-Meter yielded several surprising insights into the nature of on-device inference:
4.1 Prefill vs. Decode
In server-side deployments (e.g., using NVIDIA A100s), the autoregressive decoding phase is typically the primary bottleneck due to memory bandwidth constraints. On mobile devices, we observed the opposite: prefill is the dominant bottleneck. Processing the initial input context in a single pass places immense pressure on mobile compute units. As models scale from 70M to 1.4B parameters, decode latency increases by 10x, whereas prefill latency suffers a staggering 158x slowdown.
4.2 Architecture and Quantization
We found that model architecture often outweighs raw parameter count on the edge. For instance, the optimized SmolLM2-360M frequently outperformed the larger Pythia-1B. Additionally, we introduced the Harmonic Quantization (HQ) score to evaluate the trade-off between accuracy and latency in 4-bit quantization. Our results indicate that while quantization is highly effective for complex reasoning tasks (GSM8K), it provides diminishing returns for simpler tasks like ARC-Easy.
4.3 GPU Underutilization
Kernel-level profiling revealed that mobile GPUs are idle for approximately 21% of the time during the decoding phase. This inefficiency is largely attributed to host-side data preparation and I/O stalls, suggesting that improved CPU-GPU scheduling could yield significant performance gains.
5. Performance and Accuracy
lm-Meter demonstrates exceptional performance characteristics:
- Accuracy: It achieves ≥ 99.99% accuracy for phase-level profiling and 95-99% for kernel-level profiling compared to heavy offline tools.
- Low Overhead: Even under "Powersave" CPU configurations, the tool reduces prefill throughput by only 2.58% and decode throughput by 0.99%. In contrast, existing state-of-the-art profilers can reduce decode throughput by over 93%.
6. Conclusion and Future Work
lm-Meter provides the granular, real-time data necessary to optimize LLMs for the edge. While our current study focused on Google Pixel devices and 4-bit quantization, the framework is designed for extensibility. Future work will involve expanding the hardware scope to include NPUs and other edge platforms, integrating energy consumption profiling, and developing an "Adaptation Engine" that uses real-time telemetry to dynamically adjust model execution parameters on the fly.
Summary updated