TL;DR
GUIFlow is a training-free framework that dramatically accelerates GUI agents by treating multi-step interactions as incremental updates rather than independent inference passes. By selectively reusing and updating computational states across visual encoding, prefill, and decoding, it achieves up to a 3.49× speedup in task execution while maintaining performance comparable to standard "dense" inference.
What Problem Does This Paper Address?
AI agents powered by Vision-Language Models (VLMs) are increasingly used to navigate digital interfaces (apps, websites, desktops). However, these agents are notoriously slow. In a typical multi-step task, the agent takes a screenshot, processes it alongside a history of previous actions/screens, and decides on the next move.
The inefficiency lies in temporal redundancy. Most of the time, the screen changes only slightly between steps (e.g., a text cursor moves, or a single menu opens). Yet, standard VLM inference recomputes the entire representation of the screen and the interaction history from scratch at every single step. Existing efficiency tricks like KV-cache compression or token pruning help, but they don't address the core problem: the mismatch between continuously evolving interfaces and "start-from-zero" inference.
Methods
GUIFlow introduces a full-pipeline incremental inference paradigm. Instead of rebuilding representations, it maintains a persistent computational state across the trajectory and updates only what is necessary.
- Historical State Integration (Position-Aligned KV Injection): As the agent moves through a task, previous screenshots shift positions in the context window. GUIFlow uses "multimodal fingerprints" to identify recurring image blocks and injects their cached Key-Value (KV) states into the current context. To handle the shift in position, it applies a mathematical correction to the rotary positional embeddings (M-RoPE), ensuring the reused states remain spatially accurate.
- Current-Observation Refinement:
- Visual Encoder: It uses pixel-level change signals to identify which image patches have actually changed. Only these "active" patches are re-encoded; others reuse cached states.
- Language Model (TokenLease): Not all pixel changes require a full pass through the deep layers of the LLM. GUIFlow probes the first few layers to measure "KV deviation." If the change is minor, it "leases" the old deep-layer states; if the deviation exceeds a threshold, it triggers a refresh.
- Verified Incremental Decoding: GUI agents often produce repetitive action sequences. GUIFlow uses previous action responses as "speculative drafts." The target model verifies these drafts in a single batch pass, accepting matching tokens and only generating new ones when the draft fails. This eliminates the need for a separate, smaller "draft model."
Datasets
The authors evaluated GUIFlow across three major benchmarks to cover both fixed trajectories and live, evolving environments:
- AITW (Android in the Wild): A large-scale dataset for mobile device control.
- Mind2Web: A benchmark for generalist web agents.
- OSWorld: A complex, online closed-loop environment for desktop interaction (Ubuntu).
Key Insights and Ideas
- Representational Stability: The authors found that over 70% of observation pairs have less than 5% RGB error even across four temporal steps. More importantly, internal KV-drift maps show that internal representations are remarkably stable in regions where the UI hasn't changed.
- The "Aha" Moment on Position: You can't just copy-paste a KV cache if the token's index changes. The insight to use M-RoPE rotation to "re-align" cached states is what makes historical reuse viable beyond simple prefix matching.
- Self-Speculation: Using the agent's own history as a draft source is highly effective for GUI tasks because action structures (like JSON or specific coordinate formats) are highly predictable.
Key Results
- Speedups: GUIFlow achieved a 3.37× to 3.39× geometric-mean speedup in FLOPs across offline benchmarks. In live OSWorld tests, it reached a 3.49× speedup.
- Wall-Clock Efficiency: Beyond theoretical FLOPs, the actual processing latency was reduced by factors of 1.94× to 2.55× on NVIDIA RTX 4090 GPUs.
- Performance Retention: Despite the aggressive reuse of states, task success rates remained broadly comparable to the dense baseline. In some OSWorld scenarios, GUIFlow even slightly outperformed the baseline, likely due to the increased stability of the maintained states.
Limitations and Future Work
- Approximate vs. Exact: Incremental inference is an approximation. While the paper shows it works well in practice, it is not mathematically identical to a full recomputation, which could theoretically lead to "state drift" over extremely long trajectories.
- Memory Management: Maintaining persistent states for long trajectories increases memory pressure. The authors use a "reuse limit" ($M_{hist}$) to force periodic recomputation, but more sophisticated cache eviction strategies could be explored.
- Hardware Overhead: The gap between FLOP speedup (~3.4x) and wall-clock speedup (~2.5x) highlights the overhead of memory access and cache management. Future work could involve kernel-level optimizations to close this gap.
Summary updated