TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs

Wentao Ni, Hong Cai, Mingu Lee, Chihao Yu, Zhongming Yu, Hejia Zhang, Kangqi Zhang, Fatih Porikli, Zhijian Liu, Jishen ZhaoarXiv 2026 · arXiv

TL;DR

TileWeave is a novel system that accelerates long-context Large Language Model (LLM) prefill on mobile devices by intelligently splitting sparse attention work between the NPU and GPU. By assigning highly shared key blocks to efficient "coarse" NPU tiles and query-specific "exceptions" to the flexible GPU, it achieves up to a 3.23× speedup over state-of-the-art sparse methods while maintaining high accuracy.

What Problem Does This Paper Address?

As we push LLMs toward "long-context" applications—like summarizing entire books or acting as personal assistants with deep memory—the prefill stage becomes a massive bottleneck. Prefill is the initial phase where the model processes the entire prompt to build its internal state (the K/V cache). On mobile devices, this is constrained by high latency and limited memory bandwidth.

Block-sparse attention is a popular solution that skips unimportant token interactions to save time. However, there is a fundamental mismatch between sparse algorithms and mobile hardware:

  • Mobile NPUs (Neural Processing Units) are designed for high throughput using large, regular "tiles" of data. They are most efficient when many queries share the same set of keys.
  • Sparsity is irregular. Different queries often select different, scattered key blocks.

This leads to the "coarse-tile dilemma" (illustrated in Figure 1):

  1. Fine Tiles: Using small tiles preserves precise selections but kills NPU efficiency due to repeated data staging and coordination overhead.
  2. Coarse Tiles (Drop Exceptions): Using large tiles and dropping query-specific keys saves time but sacrifices accuracy.
  3. Coarse Tiles (Preserve Selections): Using large tiles and computing the "union" of all requested keys wastes computation on "fill-in" interactions that are eventually masked out.
Figure 1. Thin blue frames mark NPU query tiles: 64 queries in (1) and 256 in (2)–(4). Small tiles preserve selections but repeat K/V staging and coordination, limiting NPU efficiency. Coarse tiles improve reuse, but dropping exceptions risks accuracy (2), while coarse masked tiles compute unwanted interactions (3). TileWeave keeps shared keys on coarse NPU tiles and preserves exceptions through fine GPU execution (4).

Methods

TileWeave introduces heterogeneous sparse prefill to break this dilemma. Its core strategy is to partition work based on cross-query reuse:

  1. Reuse-Based Partitioning: For a group of queries, TileWeave identifies which key blocks are "highly shared" (requested by most queries in the group) and which are "exceptions" (requested by only one or two).
  2. Heterogeneous Execution:
    • NPU: Handles the shared key blocks in large, efficient coarse tiles.
  3. GPU: Handles the query-specific exceptions at a fine granularity. Since GPUs are better at irregular, indexed memory access, they can process these scattered blocks without the massive overhead an NPU would face.
  4. NPU-Native Selector: To keep the overhead of choosing which blocks to skip low, the authors designed a selector that uses "pooled keys" (averages of key blocks) and "sampled queries." This allows the selection process to run 10× faster than previous methods like MInference.
  5. Coordinated Runtime: The system overlaps NPU and GPU execution. A custom runtime merges the partial results from both engines directly into the NPU's output store using an algebraically exact online-softmax composition.
TileWeave system overview: (1) NPU-native selection, (2) partitioning by reuse, and (3) coordinated execution and merging.

Datasets

The authors evaluated TileWeave using two major LLMs: Qwen3-1.7B and Llama-3.2-3B-Instruct. They tested accuracy and performance across several benchmarks:

  • RULER: A benchmark designed to test "real" context size across 13 diverse tasks (e.g., variable tracking, multi-key retrieval).
  • LongBench v1: A comprehensive benchmark for long-context understanding, including question answering and summarization.
  • Hardware: Evaluations were performed on flagship smartphones featuring Snapdragon SM8750 and SM8850 SoCs.

Key Insights and Ideas

  • Reuse as a Granularity Criterion: The "aha moment" is realizing that cross-query reuse isn't just about saving memory—it's the perfect signal for deciding where to run a piece of code. High reuse = NPU; Low reuse = GPU.
  • Exceptions Matter: The paper proves that "low-reuse" doesn't mean "low importance." Simply dropping query-specific keys to fit NPU tiles (the "union" approach) leads to measurable accuracy drops. Preserving them on the GPU is essential for model quality.
  • Amortizing Overhead: Figure 2 shows the Hexagon NPU architecture. By using larger query tiles, TileWeave amortizes the cost of moving data from DRAM to the NPU's fast on-chip memory (VTCM).
Hexagon NPU architecture. Larger tiles help hide the latency of data movement (DMA) and coordination.

Key Results

  • Speedup: On Snapdragon phones, TileWeave is 1.35×–3.23× faster than XAttention (a leading sparse method) for full-model prefill over 1k–16k tokens. It is also up to 1.58× faster than standard dense attention.
  • Attention Latency: Looking strictly at the attention mechanism (including selection and coordination), TileWeave is up to 2.14× faster than dense attention and over 5× faster than XAttention on the SM8750.
  • Accuracy: TileWeave matches or slightly exceeds the accuracy of XAttention on RULER and stays within 0.6 points of dense attention on LongBench.

Limitations and Future Work

  • DRAM Bandwidth: Because the NPU and GPU share the same memory bus, extremely heavy GPU work can occasionally starve the NPU of bandwidth. The authors implemented a "per-tile budget" to mitigate this, but it remains a balancing act.
  • Prefill Tails: Currently, TileWeave reverts to dense attention for the very end (the "tail") of a prefill chunk if it's too small to benefit from sparsity.
  • Future Directions: The authors suggest exploring even more aggressive heterogeneous scheduling and extending the "reuse-aware" philosophy to other irregular LLM components beyond attention.

For more details, check out the full paper: "TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs."

Summary updated

Abstract

Smartphone applications increasingly require large language models to process long contexts, whereas high prefill latency and memory demands constrain local inference. Block-sparse attention reduces computation and memory traffic, but its irregular selections conflict with mobile NPUs' preference for large, regular tiles. Efficient execution must exploit cross-query reuse while avoiding redundant computation and preserving selections needed for accuracy. In this paper, we present TileWeave, which partitions sparse attention by this cross-query reuse: highly shared key blocks run in coarse NPU tiles, while low-reuse blocks run on the GPU only for the query blocks that selected them. In addition, we design an NPU-native pooled-key selector that keeps selection inexpensive, and a runtime that publishes GPU partials in the order the NPU consumes them and merges them in its existing output store. On RULER, HeteroSparse matches XAttention's accuracy with Qwen3-1.7B and improves it with Llama-3.2-3B-Instruct; its mean LongBench scores match XAttention as well. On Snapdragon SM8750 and SM8850 phones, it achieves 1.35--3.23$\times$ faster full-model prefill than XAttention over 1k--16k tokens and up to 1.58$\times$ faster than dense attention.
BibTeX
@misc{ni2026tileweave,
  author = {Wentao Ni and Hong Cai and Mingu Lee and Chihao Yu and Zhongming Yu and Hejia Zhang and Kangqi Zhang and Fatih Porikli and Zhijian Liu and Jishen Zhao},
  title = {{TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs}},
  year = {2026},
  eprint = {2611.00408},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00408},
}
Figure