Dynamic Sparse Attention on Mobile SoCs

Wangsong Yin, Daliang Xu, Mengwei Xu, Gang Huang, Xuanzhe Liu

Abstract

On-device running Large Language Models (LLMs) is nowadays a critical enabler towards preserving user privacy. We observe that the attention operator falls back from the special-purpose NPU to the general-purpose CPU/GPU because of quantization sensitivity in state-of-the-art frameworks. This fallback results in a degraded user experience and increased complexity in system scheduling. To this end, this paper presents shadowAttn, a system-algorithm codesigned sparse attention module with minimal reliance on CPU/GPU by only sparsely calculating the attention on a tiny portion of tokens. The key idea is to hide the overhead of estimating the important tokens with a NPU-based pilot compute. Further, shadowAttn proposes insightful techniques such as NPU compute graph bucketing, head-wise NPU-CPU/GPU pipeline and per-head fine-grained sparsity ratio to achieve high accuracy and efficiency. shadowAttn delivers the best performance with highly limited CPU/GPU resource; it requires much less CPU/GPU resource to deliver on-par performance of SoTA frameworks.

Introduction

The deployment of Large Language Models (LLMs) on mobile devices is essential for privacy-preserving AI. However, current mobile System-on-Chips (SoCs) face a significant hurdle: the attention operator. While Neural Processing Units (NPUs) excel at low-precision (INT8) arithmetic, the attention mechanism is highly sensitive to quantization. Standard mobile AI frameworks often experience an average accuracy drop of 18 percentage points when quantizing attention to INT8.

Consequently, frameworks "fall back" to the CPU or GPU for high-precision floating-point execution. This fallback creates performance bottlenecks and consumes significant power, often interfering with other critical system tasks. Sparse attention—calculating only the most significant tokens—offers a solution, but the computational overhead of identifying these tokens on the CPU frequently negates any efficiency gains.

Methodology: The shadowAttn Framework

We introduce shadowAttn, a system-algorithm co-designed framework that leverages the NPU as a "scout" to identify important tokens efficiently.

Pilot Compute Strategy

The fundamental insight of shadowAttn is that while the final attention calculation requires high precision, the ranking of token importance does not. shadowAttn performs an initial importance estimation (Q·K) on the NPU using INT8. This "pilot compute" identifies the top-k tokens, allowing the CPU/GPU to focus its high-precision resources only on the most relevant data.

NPU Compute Graph Bucketing

Mobile NPUs typically require static compute graphs with fixed shapes and scales. However, LLM activation values vary significantly. To address this, shadowAttn employs a "bucketing" technique to handle dynamic scaling factors without re-compilation.

Workflow of shadowAttn's NPU compute graph bucketing and execution stages.

As shown in the figure above, the system pre-builds a bucket of multiple graphs with varying scaling factors during an offline stage. At runtime, the system dynamically selects the most appropriate graph to maintain accuracy.

Head-wise Pipelining and Sparsity

To maximize hardware utilization, shadowAttn overlaps NPU estimation with CPU sparse computation. By scheduling attention "heads" across processors, the system minimizes idle time. Furthermore, shadowAttn applies per-head fine-grained sparsity. Since not all attention heads contribute equally to model performance, less critical heads are assigned higher sparsity ratios, optimizing the balance between accuracy and speed.

Evaluation

We evaluated shadowAttn using Qwen2 and PhoneLM models (0.5B and 1.5B parameters) across several tasks: ArxivSum (summarization), DroidCall (GUI automation), and Octopus (API function calling). Profiling was conducted using 128 samples from the WikiText-2 corpus.

Key Results

shadowAttn demonstrates significant improvements in both performance and efficiency:

  • Performance: shadowAttn achieves up to a 6.9x speedup at the kernel level and a 4.5x end-to-end speedup compared to standard CPU-based floating-point attention.
  • Energy Efficiency: By offloading the majority of the workload to the NPU, energy consumption is reduced by up to 7.7x.
  • Accuracy Preservation: Unlike naive quantization which causes massive accuracy loss, shadowAttn maintains high fidelity, with an average drop of only 0.4 percentage points.
Latency breakdown and accuracy comparison for shadowAttn.

The results in the figure above highlight the effectiveness of token-level sparsity in reducing latency while maintaining superior accuracy compared to block-based methods.

Conclusion and Future Work

shadowAttn effectively bridges the gap between NPU efficiency and attention precision requirements. While it requires an offline profiling stage, the runtime benefits for mobile LLM inference are substantial. Future work will explore extending these optimizations to the memory-bound decoding stage and evaluating performance across a broader range of mobile SoC architectures.

Summary updated

Abstract

On-device running Large Language Models (LLMs) is nowadays a critical enabler towards preserving user privacy. We observe that the attention operator falls back from the special-purpose NPU to the general-purpose CPU/GPU because of quantization sensitivity in state-of-the-art frameworks. This fallback results in a degraded user experience and increased complexity in system scheduling. To this end, this paper presents shadowAttn, a system-algorithm codesigned sparse attention module with minimal reliance on CPU/GPU by only sparsely calculating the attention on a tiny portion of tokens. The key idea is to hide the overhead of estimating the important tokens with a NPU-based pilot compute. Further, shadowAttn proposes insightful techniques such as NPU compute graph bucketing, head-wise NPU-CPU/GPU pipeline and per-head fine-grained sparsity ratio to achieve high accuracy and efficiency. shadowAttn delivers the best performance with highly limited CPU/GPU resource; it requires much less CPU/GPU resource to deliver on-par performance of SoTA frameworks.
BibTeX
@misc{yin2025dynamic,
  author = {Wangsong Yin and Daliang Xu and Mengwei Xu and Gang Huang and Xuanzhe Liu},
  title = {{Dynamic Sparse Attention on Mobile SoCs}},
  year = {2025},
  eprint = {2508.16703},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.16703},
}
Figure