TL;DR
TileWeave is a novel system that accelerates long-context Large Language Model (LLM) prefill on mobile devices by intelligently splitting sparse attention work between the NPU and GPU. By assigning highly shared key blocks to efficient "coarse" NPU tiles and query-specific "exceptions" to the flexible GPU, it achieves up to a 3.23× speedup over state-of-the-art sparse methods while maintaining high accuracy.
What Problem Does This Paper Address?
As we push LLMs toward "long-context" applications—like summarizing entire books or acting as personal assistants with deep memory—the prefill stage becomes a massive bottleneck. Prefill is the initial phase where the model processes the entire prompt to build its internal state (the K/V cache). On mobile devices, this is constrained by high latency and limited memory bandwidth.
Block-sparse attention is a popular solution that skips unimportant token interactions to save time. However, there is a fundamental mismatch between sparse algorithms and mobile hardware:
- Mobile NPUs (Neural Processing Units) are designed for high throughput using large, regular "tiles" of data. They are most efficient when many queries share the same set of keys.
- Sparsity is irregular. Different queries often select different, scattered key blocks.
This leads to the "coarse-tile dilemma" (illustrated in Figure 1):
- Fine Tiles: Using small tiles preserves precise selections but kills NPU efficiency due to repeated data staging and coordination overhead.
- Coarse Tiles (Drop Exceptions): Using large tiles and dropping query-specific keys saves time but sacrifices accuracy.
- Coarse Tiles (Preserve Selections): Using large tiles and computing the "union" of all requested keys wastes computation on "fill-in" interactions that are eventually masked out.
Methods
TileWeave introduces heterogeneous sparse prefill to break this dilemma. Its core strategy is to partition work based on cross-query reuse:
- Reuse-Based Partitioning: For a group of queries, TileWeave identifies which key blocks are "highly shared" (requested by most queries in the group) and which are "exceptions" (requested by only one or two).
- Heterogeneous Execution:
- NPU: Handles the shared key blocks in large, efficient coarse tiles.
- GPU: Handles the query-specific exceptions at a fine granularity. Since GPUs are better at irregular, indexed memory access, they can process these scattered blocks without the massive overhead an NPU would face.
- NPU-Native Selector: To keep the overhead of choosing which blocks to skip low, the authors designed a selector that uses "pooled keys" (averages of key blocks) and "sampled queries." This allows the selection process to run 10× faster than previous methods like MInference.
- Coordinated Runtime: The system overlaps NPU and GPU execution. A custom runtime merges the partial results from both engines directly into the NPU's output store using an algebraically exact online-softmax composition.
Datasets
The authors evaluated TileWeave using two major LLMs: Qwen3-1.7B and Llama-3.2-3B-Instruct. They tested accuracy and performance across several benchmarks:
- RULER: A benchmark designed to test "real" context size across 13 diverse tasks (e.g., variable tracking, multi-key retrieval).
- LongBench v1: A comprehensive benchmark for long-context understanding, including question answering and summarization.
- Hardware: Evaluations were performed on flagship smartphones featuring Snapdragon SM8750 and SM8850 SoCs.
Key Insights and Ideas
- Reuse as a Granularity Criterion: The "aha moment" is realizing that cross-query reuse isn't just about saving memory—it's the perfect signal for deciding where to run a piece of code. High reuse = NPU; Low reuse = GPU.
- Exceptions Matter: The paper proves that "low-reuse" doesn't mean "low importance." Simply dropping query-specific keys to fit NPU tiles (the "union" approach) leads to measurable accuracy drops. Preserving them on the GPU is essential for model quality.
- Amortizing Overhead: Figure 2 shows the Hexagon NPU architecture. By using larger query tiles, TileWeave amortizes the cost of moving data from DRAM to the NPU's fast on-chip memory (VTCM).
Key Results
- Speedup: On Snapdragon phones, TileWeave is 1.35×–3.23× faster than XAttention (a leading sparse method) for full-model prefill over 1k–16k tokens. It is also up to 1.58× faster than standard dense attention.
- Attention Latency: Looking strictly at the attention mechanism (including selection and coordination), TileWeave is up to 2.14× faster than dense attention and over 5× faster than XAttention on the SM8750.
- Accuracy: TileWeave matches or slightly exceeds the accuracy of XAttention on RULER and stays within 0.6 points of dense attention on LongBench.
Limitations and Future Work
- DRAM Bandwidth: Because the NPU and GPU share the same memory bus, extremely heavy GPU work can occasionally starve the NPU of bandwidth. The authors implemented a "per-tile budget" to mitigate this, but it remains a balancing act.
- Prefill Tails: Currently, TileWeave reverts to dense attention for the very end (the "tail") of a prefill chunk if it's too small to benefit from sparsity.
- Future Directions: The authors suggest exploring even more aggressive heterogeneous scheduling and extending the "reuse-aware" philosophy to other irregular LLM components beyond attention.
For more details, check out the full paper: "TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs."
Summary updated