TL;DR
Tandem is a specialized runtime designed to serve hybrid linear-attention GUI agents efficiently on mobile Neural Processing Units (NPUs). By optimizing recurrent-state checkpointing, speculative decoding kernels, and NPU power management, it reduces per-step latency by up to 30% and energy consumption by 33% on modern Snapdragon hardware.
What Problem Does This Paper Address?
Smartphone GUI agents—AI assistants that "see" your screen and "act" on your behalf—operate in a loop: observe the screen, decide on an action (tap, scroll, type), and wait for the app to respond. This creates a unique workload for Large Language Models (LLMs).
While hybrid linear-attention models (like Qwen3.5-4B) are memory-efficient for mobile use, they face three major hurdles on NPUs:
- Inefficient Context Reuse: Standard LLM serving saves "checkpoints" at message boundaries. However, GUI agent prompts often rewrite recent history or screen descriptions in the middle of a prompt, rendering these standard checkpoints useless and forcing the NPU to re-process hundreds of tokens.
- Speculative Decoding Overhead: Speculative decoding (proposing multiple tokens at once) requires "rollback states" to undo incorrect guesses. On NPUs, generating these states often forces the entire prompt to be processed on slower vector units rather than fast matrix units, creating a "rollback tax" that outweighs the speedup from speculation.
- Idle Power Drain: Keeping a large model loaded in memory often keeps the NPU "awake" at high power levels, even during the long pauses between user requests.
Methods
The authors introduce Tandem, which implements several novel optimizations within the llama.cpp framework:
- Divergence-Point Checkpoints: Instead of arbitrary boundaries, Tandem saves the recurrent state exactly where a new prompt begins to differ from the previous one. This maximizes prefix reuse even when history is rewritten.
- Rollback-Aware Prefill: To support speculative decoding without the "rollback tax," Tandem uses the NPU's high-speed matrix unit (HMX) for the bulk of the prompt and only switches to the per-token vector unit (HVX) for a tiny "tail" of the last few tokens to generate rollback states.
- Column-Shared Verification: A custom NPU kernel that reads model weights once and applies them to all proposed tokens in a speculative batch, significantly reducing memory traffic.
- Idle Power Release: Tandem releases NPU "power votes" (requests for high clocks/voltage) during idle intervals without unloading the model from RAM. This allows the NPU to enter low-power states while keeping the agent ready for a near-instant restart.
Datasets
The authors evaluated Tandem using several GUI-focused datasets and real-world traces:
- App Set: 21 Chinese apps with text and screenshot-based requests.
- Replay Trace: 61 sequential requests from recorded agent episodes to test prefix reuse.
- Held-out Set: 510 steps from 54 episodes across 12 apps.
- AndroidControl: 300 public test steps from 56 episodes.
- On-Device Tasks: 100 real-time tasks performed in 17 apps installed on the test phone.
Key Insights and Ideas
- The "Rollback Tax" is Real: The authors discovered that enabling standard speculative decoding on a hybrid model actually made it slower (from 1.85s to 2.52s per step) because of how the NPU handles state snapshots. Tandem's "short tail" approach is the "aha moment" that makes speculation profitable on mobile hardware.
- Locality of Divergence: In GUI tasks, prompts don't just grow; they evolve. By tracking where prompts diverge, Tandem achieves context reuse nearly identical to full-attention models but with the memory benefits of recurrent layers.
- Power Management vs. Model Loading: You don't have to unload a 4GB model to save battery. Releasing NPU session votes can drop standby power by over 70% while adding only a negligible (2-4ms) delay to the next request.
Key Results
Evaluated on the Snapdragon 8 Elite (OnePlus 13T):
- Latency: Reduced text-step latency by 30% (from 1.96s to 1.38s) and screenshot-step latency by 16%.
- Energy: Reduced energy consumption per text step by 33%.
- Standby Power: Lowered power draw with the model loaded from 0.76W to 0.22W, effectively eliminating the "residency tax" of keeping the NPU awake.
- Efficiency: Tandem is 3.6x–4.0x faster than the best GPU-based mobile inference engines (like MNN) for these hybrid models.
Limitations and Future Work
- Screenshot Bottleneck: Even with Tandem, screenshot-based steps are significantly slower than text-based ones due to the vision encoder. The authors suggest that future work could adapt the image-token budget dynamically based on screen content.
- Hardware Specificity: The current implementation is highly optimized for the Qualcomm Hexagon NPU architecture. Porting these gains to other mobile accelerators (like MediaTek's APU or Apple's Neural Engine) would require rewriting the matrix and vector kernels.
- Image Budget Tradeoffs: While reducing image tokens speeds up the model, it can impact accuracy on screens requiring fine visual detail.
The project is available at https://github.com/flyingdoog/tandem.
Summary updated