Tandem: Serving Hybrid Linear-Attention GUI Agents on Mobile NPUs

Zhuomin Chen, Jingchao Ni, Xu Zheng, Yong Chen, Mo Sha, Wei Cheng, Dongsheng LuoarXiv 2026 · arXiv

TL;DR

Tandem is a specialized runtime designed to serve hybrid linear-attention GUI agents efficiently on mobile Neural Processing Units (NPUs). By optimizing recurrent-state checkpointing, speculative decoding kernels, and NPU power management, it reduces per-step latency by up to 30% and energy consumption by 33% on modern Snapdragon hardware.

What Problem Does This Paper Address?

Smartphone GUI agents—AI assistants that "see" your screen and "act" on your behalf—operate in a loop: observe the screen, decide on an action (tap, scroll, type), and wait for the app to respond. This creates a unique workload for Large Language Models (LLMs).

While hybrid linear-attention models (like Qwen3.5-4B) are memory-efficient for mobile use, they face three major hurdles on NPUs:

  1. Inefficient Context Reuse: Standard LLM serving saves "checkpoints" at message boundaries. However, GUI agent prompts often rewrite recent history or screen descriptions in the middle of a prompt, rendering these standard checkpoints useless and forcing the NPU to re-process hundreds of tokens.
  2. Speculative Decoding Overhead: Speculative decoding (proposing multiple tokens at once) requires "rollback states" to undo incorrect guesses. On NPUs, generating these states often forces the entire prompt to be processed on slower vector units rather than fast matrix units, creating a "rollback tax" that outweighs the speedup from speculation.
  3. Idle Power Drain: Keeping a large model loaded in memory often keeps the NPU "awake" at high power levels, even during the long pauses between user requests.

Methods

The authors introduce Tandem, which implements several novel optimizations within the llama.cpp framework:

  • Divergence-Point Checkpoints: Instead of arbitrary boundaries, Tandem saves the recurrent state exactly where a new prompt begins to differ from the previous one. This maximizes prefix reuse even when history is rewritten.
  • Rollback-Aware Prefill: To support speculative decoding without the "rollback tax," Tandem uses the NPU's high-speed matrix unit (HMX) for the bulk of the prompt and only switches to the per-token vector unit (HVX) for a tiny "tail" of the last few tokens to generate rollback states.
  • Column-Shared Verification: A custom NPU kernel that reads model weights once and applies them to all proposed tokens in a speculative batch, significantly reducing memory traffic.
  • Idle Power Release: Tandem releases NPU "power votes" (requests for high clocks/voltage) during idle intervals without unloading the model from RAM. This allows the NPU to enter low-power states while keeping the agent ready for a near-instant restart.
Figure 4. Tandem’s serving architecture, showing cross-request recurrent checkpoints, temporary rollback states for speculative decoding, and idle NPU session control.

Datasets

The authors evaluated Tandem using several GUI-focused datasets and real-world traces:

  • App Set: 21 Chinese apps with text and screenshot-based requests.
  • Replay Trace: 61 sequential requests from recorded agent episodes to test prefix reuse.
  • Held-out Set: 510 steps from 54 episodes across 12 apps.
  • AndroidControl: 300 public test steps from 56 episodes.
  • On-Device Tasks: 100 real-time tasks performed in 17 apps installed on the test phone.

Key Insights and Ideas

  • The "Rollback Tax" is Real: The authors discovered that enabling standard speculative decoding on a hybrid model actually made it slower (from 1.85s to 2.52s per step) because of how the NPU handles state snapshots. Tandem's "short tail" approach is the "aha moment" that makes speculation profitable on mobile hardware.
  • Locality of Divergence: In GUI tasks, prompts don't just grow; they evolve. By tracking where prompts diverge, Tandem achieves context reuse nearly identical to full-attention models but with the memory benefits of recurrent layers.
  • Power Management vs. Model Loading: You don't have to unload a 4GB model to save battery. Releasing NPU session votes can drop standby power by over 70% while adding only a negligible (2-4ms) delay to the next request.
Figure 3. New prompt tokens per request of the replay trace: the app set, then five recorded episodes (dot- ted lines mark episode starts). The full-attention Qwen3-VL-4B runs on the same engine. Divergence-point checkpoints (§4.2) close most of the checkpoint gap.

Key Results

Evaluated on the Snapdragon 8 Elite (OnePlus 13T):

  • Latency: Reduced text-step latency by 30% (from 1.96s to 1.38s) and screenshot-step latency by 16%.
  • Energy: Reduced energy consumption per text step by 33%.
  • Standby Power: Lowered power draw with the model loaded from 0.76W to 0.22W, effectively eliminating the "residency tax" of keeping the NPU awake.
  • Efficiency: Tandem is 3.6x–4.0x faster than the best GPU-based mobile inference engines (like MNN) for these hybrid models.
Figure 2. Server time by phase on the app set, with stock checkpoint and batching policies, Tandem’s specu- lative NPU kernels and fan cooling. Phase times cover text requests 2–21 and screenshot requests 4–21; totals are 1.31 and 5.39 s.

Limitations and Future Work

  • Screenshot Bottleneck: Even with Tandem, screenshot-based steps are significantly slower than text-based ones due to the vision encoder. The authors suggest that future work could adapt the image-token budget dynamically based on screen content.
  • Hardware Specificity: The current implementation is highly optimized for the Qualcomm Hexagon NPU architecture. Porting these gains to other mobile accelerators (like MediaTek's APU or Apple's Neural Engine) would require rewriting the matrix and vector kernels.
  • Image Budget Tradeoffs: While reducing image tokens speeds up the model, it can impact accuracy on screens requiring fine visual detail.

The project is available at https://github.com/flyingdoog/tandem.

Summary updated

Abstract

Smartphone GUI agents complete tasks in mobile apps by repeatedly reading the screen and asking a model for the next action. Mobile neural processing units (NPUs) enable local execution, keeping screen contents on the phone, but model latency accumulates with every action. Hybrid linear attention models reduce memory requirements, yet reusing their recurrent state requires checkpoints at the appropriate input positions. Agent requests retain instructions but rewrite recent history, so checkpoint policies designed for chat continuation can miss reusable prefixes. Retaining rollback states for speculative decoding can also move prompt processing off the NPU's matrix unit, adding overhead that outweighs decoding savings on short action outputs. A loaded model can further hold the NPU awake between tasks. We present Tandem, a runtime for hybrid GUI agents on mobile NPUs. Tandem saves recurrent state where consecutive prompts diverge, allowing later requests to reuse previously processed context. It keeps most prefill on the matrix unit and produces rollback states through a short per-token tail. Its verification kernel reads each weight block once for all proposed tokens. Between request bursts, Tandem releases NPU power votes without unloading the model. We implement Tandem in llama.cpp and evaluate it on recorded agent interactions across two Snapdragon generations. On a Snapdragon 8 Elite, Tandem reduces per-step latency by 30% for text-based screen descriptions and 16% for screenshots relative to llama.cpp's fastest NPU configuration, reduces energy per text step by 33%, and lowers standby power with the model loaded from 0.76 to 0.22 W.
BibTeX
@misc{chen2026tandem,
  author = {Zhuomin Chen and Jingchao Ni and Xu Zheng and Yong Chen and Mo Sha and Wei Cheng and Dongsheng Luo},
  title = {{Tandem: Serving Hybrid Linear-Attention GUI Agents on Mobile NPUs}},
  year = {2026},
  eprint = {2610.26660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.26660},
}
Figure