GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents

Ye Li, Haoran Ding, Huanan Liu, Yuan Meng, Yutian Shen, Yuansong Wang, Kangye Ji, Hanyun Cui, Hongbo Duan, Chen Tang, Jiawei Li, Shu-Tao Xia, Zhi WangarXiv 2026 · arXiv

TL;DR

GUIFlow is a training-free framework that dramatically accelerates GUI agents by treating multi-step interactions as incremental updates rather than independent inference passes. By selectively reusing and updating computational states across visual encoding, prefill, and decoding, it achieves up to a 3.49× speedup in task execution while maintaining performance comparable to standard "dense" inference.

What Problem Does This Paper Address?

AI agents powered by Vision-Language Models (VLMs) are increasingly used to navigate digital interfaces (apps, websites, desktops). However, these agents are notoriously slow. In a typical multi-step task, the agent takes a screenshot, processes it alongside a history of previous actions/screens, and decides on the next move.

The inefficiency lies in temporal redundancy. Most of the time, the screen changes only slightly between steps (e.g., a text cursor moves, or a single menu opens). Yet, standard VLM inference recomputes the entire representation of the screen and the interaction history from scratch at every single step. Existing efficiency tricks like KV-cache compression or token pruning help, but they don't address the core problem: the mismatch between continuously evolving interfaces and "start-from-zero" inference.

Methods

GUIFlow introduces a full-pipeline incremental inference paradigm. Instead of rebuilding representations, it maintains a persistent computational state across the trajectory and updates only what is necessary.

  1. Historical State Integration (Position-Aligned KV Injection): As the agent moves through a task, previous screenshots shift positions in the context window. GUIFlow uses "multimodal fingerprints" to identify recurring image blocks and injects their cached Key-Value (KV) states into the current context. To handle the shift in position, it applies a mathematical correction to the rotary positional embeddings (M-RoPE), ensuring the reused states remain spatially accurate.
  2. Current-Observation Refinement:
    • Visual Encoder: It uses pixel-level change signals to identify which image patches have actually changed. Only these "active" patches are re-encoded; others reuse cached states.
  3. Language Model (TokenLease): Not all pixel changes require a full pass through the deep layers of the LLM. GUIFlow probes the first few layers to measure "KV deviation." If the change is minor, it "leases" the old deep-layer states; if the deviation exceeds a threshold, it triggers a refresh.
  4. Verified Incremental Decoding: GUI agents often produce repetitive action sequences. GUIFlow uses previous action responses as "speculative drafts." The target model verifies these drafts in a single batch pass, accepting matching tokens and only generating new ones when the draft fails. This eliminates the need for a separate, smaller "draft model."
Overview of the GUIFlow framework showing how it integrates historical states, refines current observations, and verifies decoding.

Datasets

The authors evaluated GUIFlow across three major benchmarks to cover both fixed trajectories and live, evolving environments:

  • AITW (Android in the Wild): A large-scale dataset for mobile device control.
  • Mind2Web: A benchmark for generalist web agents.
  • OSWorld: A complex, online closed-loop environment for desktop interaction (Ubuntu).

Key Insights and Ideas

  • Representational Stability: The authors found that over 70% of observation pairs have less than 5% RGB error even across four temporal steps. More importantly, internal KV-drift maps show that internal representations are remarkably stable in regions where the UI hasn't changed.
  • The "Aha" Moment on Position: You can't just copy-paste a KV cache if the token's index changes. The insight to use M-RoPE rotation to "re-align" cached states is what makes historical reuse viable beyond simple prefix matching.
  • Self-Speculation: Using the agent's own history as a draft source is highly effective for GUI tasks because action structures (like JSON or specific coordinate formats) are highly predictable.
Analysis of incremental changes in GUI observations and internal representations, showing high stability over time.

Key Results

  • Speedups: GUIFlow achieved a 3.37× to 3.39× geometric-mean speedup in FLOPs across offline benchmarks. In live OSWorld tests, it reached a 3.49× speedup.
  • Wall-Clock Efficiency: Beyond theoretical FLOPs, the actual processing latency was reduced by factors of 1.94× to 2.55× on NVIDIA RTX 4090 GPUs.
  • Performance Retention: Despite the aggressive reuse of states, task success rates remained broadly comparable to the dense baseline. In some OSWorld scenarios, GUIFlow even slightly outperformed the baseline, likely due to the increased stability of the maintained states.
Qualitative comparison of task execution time on an OSWorld task, showing GUIFlow is 2.16x faster than the baseline.

Limitations and Future Work

  • Approximate vs. Exact: Incremental inference is an approximation. While the paper shows it works well in practice, it is not mathematically identical to a full recomputation, which could theoretically lead to "state drift" over extremely long trajectories.
  • Memory Management: Maintaining persistent states for long trajectories increases memory pressure. The authors use a "reuse limit" ($M_{hist}$) to force periodic recomputation, but more sophisticated cache eviction strategies could be explored.
  • Hardware Overhead: The gap between FLOP speedup (~3.4x) and wall-clock speedup (~2.5x) highlights the overhead of memory access and cache management. Future work could involve kernel-level optimizations to close this gap.

Summary updated

Abstract

GUI agents built on vision-language models interact with continuously evolving interfaces, yet conventional inference repeatedly recomputes representations of largely overlapping observations and histories. This mismatch makes multi-step interaction costly, while changes in context, interface content, and action arguments prevent straightforward cache reuse. We introduce GUIFlow, a framework for full-pipeline incremental inference in multi-image GUI interaction. Our central insight is that advancing a decision need not require rebuilding all supporting representations: each interaction step can instead selectively update computational states maintained across the trajectory. To support new decisions as the context evolves, GUIFlow adapts how past computation is reused: historical KV states are position-aligned and injected into the current context; change signals guide selective updates of current-observation states in the vision encoder and language model; and earlier action responses serve as speculative drafts verified by the target model. Together, these mechanisms enable cross-step reuse throughout visual encoding, prefill, and decoding without additional training, a separate draft model, or changes to the model-visible inputs and action interface. Experiments demonstrate up to 3.39x geometric-mean speedup across offline multi-image benchmarks and up to 3.49x in online, closed-loop interaction on OSWorld, while maintaining task performance broadly comparable to the dense baseline.
BibTeX
@misc{li2026guiflow,
  author = {Ye Li and Haoran Ding and Huanan Liu and Yuan Meng and Yutian Shen and Yuansong Wang and Kangye Ji and Hanyun Cui and Hongbo Duan and Chen Tang and Jiawei Li and Shu-Tao Xia and Zhi Wang},
  title = {{GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents}},
  year = {2026},
  eprint = {2610.34959},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34959},
}
Figure