GPA: Learning GUI Process Automation from Demonstrations

Zirui Zhao, Jun Hao Liew, Yan Yang, Wenzhuo Yang, Ziyang Luo, Doyen Sahoo, Silvio Savarese, Junnan LiarXiv 2026 · arXiv

TL;DR

GPA (GUI Process Automation) is a vision-based automation system that learns to navigate complex software interfaces from a single user demonstration. By representing user interfaces as spatial graphs and using Sequential Monte Carlo (SMC) for localization, it achieves 100% success rates on enterprise tasks while running 10x faster and more reliably than cloud-based Vision Language Model (VLM) agents.

What Problem Does This Paper Address?

Automating repetitive tasks on Graphical User Interfaces (GUIs) is a cornerstone of enterprise productivity, but current solutions fall into two problematic camps:

  1. Traditional RPA (Robotic Process Automation): These systems are notoriously brittle. They rely on rigid metadata (like HTML tags) or fixed coordinates. If a website layout shifts slightly or a user changes their screen resolution, the automation breaks, leading to high maintenance costs.
  2. VLM-based GUI Agents: Modern AI agents (like those powered by GPT-4V or Gemini) are flexible but non-deterministic. They might perform a task correctly nine times and "hallucinate" an error on the tenth. Furthermore, they suffer from high latency and privacy risks because they must stream sensitive screenshots to external cloud APIs.

Salesforce researchers developed GPA to find the "Goldilocks" zone: a system as easy to teach as a VLM agent but as fast, private, and reliable as traditional RPA.

Methods

GPA operates in two distinct phases: Demonstration and Execution.

  • Demonstration Phase: A user performs a task once. The system records (screenshot, action) pairs and parses them into a UI Graph. In this graph, nodes represent UI elements (icons, text) containing visual embeddings and OCR data, while edges connect spatially nearby elements.
  • Execution Phase: To replay the task, GPA must solve the "UI Grounding" problem—finding the right button to click even if the window has been resized or the layout has shifted.
Figure 1: Two phases of GPA: demonstration phase (a) and execution phase (b). In the demonstration phase, a user interacts with the local desktop environment while the recorder captures screenshots and actions, parses them into step subgraphs, and applies LLM-based post-processing for cleanup and variable extraction. In the execution phase, GPA receives the latest observation from the local computer, loads the current recorded step, parses the screenshot into a UI graph, and combines the parsed graph, recorded workflow graph, and action in SMC localization. A finite state machine orchestrates all the components, checking readiness of each step, control execution, and handle errors.

The core innovation is treating grounding as a Sequential Monte Carlo (SMC) localization problem. Instead of just looking for a specific button, GPA looks at the context—the surrounding text and icons. If the target button's appearance changes or it's missed by the detector, the system uses the relative positions of stable neighboring nodes to infer the target's location. This mimics how a human finds a "Submit" button by looking for the text fields next to it.

To ensure reliability, GPA uses a Readiness Calibration mechanism. It only executes an action if its confidence score (derived from the SMC particle distribution) exceeds a strict threshold. If the screen isn't ready (e.g., a page is still loading), the system waits and retries rather than guessing.

Datasets

The researchers evaluated GPA using a pilot study of 16 desktop GUI tasks categorized by difficulty:

  • Simple tasks (avg. 10.8 steps): Drafting emails, downloading receipts, flight searching.
  • Hard tasks (avg. 27.27 steps): Google Calendar events, SAP ERP form-filling, and complex HR workflows.

For the underlying vision components, they utilized a fine-tuned OmniParser for element detection and IconCLIP for generating visual embeddings of UI components.

Key Insights and Ideas

  • Context-Aware Grounding: By representing the UI as a graph (as seen in Figure 2), the system becomes robust to "UI drift." Even if a specific icon changes, the spatial relationship to nearby text usually remains constant.
  • Local-First Architecture: GPA uses "tiny" local models rather than massive cloud LLMs. This eliminates network latency and ensures that sensitive enterprise data never leaves the local machine.
  • Deterministic Execution: Unlike generative agents that predict the "next token," GPA follows a structured workflow template. This makes the automation predictable, auditable, and easier to constrain for mission-critical enterprise use.
Example of a UI graph where the target node (highlighted) is localized using its spatial neighbors.

Key Results

In head-to-head tests against a high-end VLM baseline (Gemini 3 Pro with computer-use tools), GPA showed significant advantages:

  • Success Rate: GPA achieved a 100% success rate across all tasks, whereas the Gemini baseline dropped to ~87% on longer, "hard" tasks due to compounding errors.
  • Speed: GPA was roughly 10x faster, completing tasks in an average of 33.7 seconds compared to 329.3 seconds for the cloud-based agent.
  • Robustness: The SMC-based localization successfully handled window rescaling and layout changes that typically break traditional RPA scripts.

Limitations and Future Work

The authors acknowledge that GPA is a record-and-replay system, not a reasoning engine. It cannot exercise judgment—for example, it can't navigate a calendar to pick a specific date unless that exact date was used in the demonstration.

Future work aims to combine GPA with LLM agents to create "self-healing" workflows. In this hybrid model, an LLM would handle the high-level reasoning and error recovery, while GPA handles the fast, reliable execution of the GUI interactions. The researchers are also exploring "precondition tracking" to ensure a tool is only called when the software environment is in the correct state.

Summary updated

Abstract

GUI Process Automation (GPA) is a lightweight but general vision-based Robotic Process Automation (RPA), which enables fast and stable process replay with only a single demo. Addressing the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents, GPA introduces three core benefits: (1) Robustness via Sequential Monte Carlo-based localization to handle rescaling and detection uncertainty; (2) Deterministic and Reliability safeguarded by readiness calibration; and (3) Privacy through fast, fully local execution. This approach delivers the adaptability, robustness, and security required for enterprise workflows. It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution. We conducted a pilot experiment to compare GPA with Gemini 3 Pro (with CUA tools) and found that GPA achieves higher success rate with 10 times faster execution speed in finishing long-horizon GUI tasks.
BibTeX
@misc{zhao2026gpa,
  author = {Zirui Zhao and Jun Hao Liew and Yan Yang and Wenzhuo Yang and Ziyang Luo and Doyen Sahoo and Silvio Savarese and Junnan Li},
  title = {{GPA: Learning GUI Process Automation from Demonstrations}},
  year = {2026},
  eprint = {2604.01676},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.01676},
}
Figure