From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation

Zezhou Wang, Ziyun Zhang, Xiaoyi Zhang, Sanglu Lu, Yan Lu

Abstract

Vision-language models are increasingly deployed as computer-use agents (CUAs) that operate desktops and browsers. Top-performing CUAs are framework-based systems that decompose planning and execution, while end-to-end screenshot-to-action policies are easier to deploy but lag behind on benchmarks such as OSWorld-Verified. GUI datasets like OSWorld pose two bottlenecks: they expose only a few hundred interactive, verifiable tasks and environments, and expert trajectories must be gathered by interacting with these environments, making such data hard to scale. We therefore ask how reinforcement learning from verifiable rewards (RLVR) can best exploit a small pool of exist expert trajectories to train end-to-end policies. Naively mixing these off-policy traces into on-policy RLVR is brittle: even after format conversion, expert trajectories exhibit structural mismatch and distribution shift from the learner. We propose BEPA (Bi-Level Expert-to-Policy Assimilation), which turns static expert traces into policy-aligned guidance via self-rolled reachable trajectories under the base policy (LEVEL-1) and a per-task, dynamically updated cache used in RLVR (LEVEL-2). On OSWorld-Verified, BEPA improves UITARS1.5-7B success from 22.87% to 32.13% and raises a held-out split from 5.74% to 10.30%, with consistent gains on MMBench-GUI and Online-Mind2Web.

Introduction

The development of Computer-Use Agents (CUAs) capable of navigating complex graphical user interfaces (GUIs) has diverged into two primary architectures. Framework-based systems utilize a modular stack of planners, executors, and grounders to achieve high performance. However, these systems are often cumbersome to deploy and computationally expensive. Conversely, end-to-end (E2E) models, which map screenshots directly to actions, offer a more streamlined and faster alternative but significantly lag in accuracy on benchmarks like OSWorld.

A natural approach to bridge this gap is to leverage high-quality trajectories from framework-based "experts" to train E2E "student" models. However, direct imitation often fails due to two critical mismatches. First, a structural mismatch exists because framework agents use internal APIs or multi-step reasoning that an E2E model cannot replicate from a single screenshot. Second, a distribution gap occurs because expert trajectories are often "off-manifold" for the learner; forcing a model to imitate actions far beyond its current capability leads to optimization instability and "exploration collapse."

Methodology: Bi-Level Expert-to-Policy Assimilation (BEPA)

To address these challenges, we introduce BEPA, a framework designed to transform "alien" expert data into "policy-aligned" guidance through two complementary levels.

Overview of the BEPA framework showing LEVEL-1 (Init Phase) and LEVEL-2 (Training Phase).

LEVEL-1: Self-Rolled Execution

Instead of mandating that the model replicate the expert's exact actions, LEVEL-1 focuses on plan extraction. A high-level, step-by-step plan is extracted from the expert trajectory. The base model then attempts to execute this plan within the environment. If the model successfully completes the task, the resulting trajectory is saved. Because these actions are generated by the model itself, the data is inherently "reachable" and aligned with the model's own distribution, effectively bridging the distribution gap.

LEVEL-2: Self-Aligned Off-Policy Assimilation

During the Reinforcement Learning phase (utilizing Group Relative Policy Optimization, or GRPO), BEPA maintains a per-task cache of successful trajectories. This level involves two key mechanisms:

  1. Injection: If the agent fails all its current on-policy attempts for a specific task, it "borrows" a successful trajectory from the cache. This ensures the optimizer receives a positive signal even during difficult exploration phases.
  2. Dynamic Update: As the agent's performance improves, it replaces the initial expert-derived trajectories in the cache with its own most recent on-policy successes. This ensures that the guidance evolves alongside the learner, maintaining policy alignment.
Illustration of the structural mismatch between framework agents and E2E models, and how conversion/self-rolling mitigates this.

Experiments and Results

We evaluated BEPA across several challenging benchmarks, including OSWorld-Verified (369 real-world desktop tasks), MMBench-GUI (for grounding and understanding), and Online-Mind2Web (for live web navigation).

Performance Gains

On OSWorld-Verified, BEPA significantly boosted the success rate of the UITARS1.5-7B model from 22.87% to 32.13%, representing a ~40% relative improvement. Crucially, the model demonstrated strong generalization; on a strictly held-out split of tasks, success rates increased from 5.74% to 10.30%.

Comparative Analysis

BEPA outperformed standard RL methods like GRPO and other expert-integration techniques such as LUFFY or simple Supervised Fine-Tuning (SFT) followed by RL. These baseline methods often suffered from performance degradation or brittleness when faced with the dynamic nature of GUI environments. Furthermore, BEPA proved highly data-efficient, achieving these gains using a small pool of only 115 successful expert trajectories.

Discussion and Conclusion

The success of BEPA highlights the importance of "scaffolding over imitation." By using expert plans to guide exploration rather than expert actions to define the loss, the model learns to solve tasks in its own "voice." The dynamic cache mechanism prevents the "forgetting" common in static SFT by keeping training data synchronized with the model's current state.

While BEPA shows great promise for desktop and web agents, future work will explore its application to mobile platforms and environments with noisier, preference-based feedback. Additionally, optimizing the computational overhead of the self-rolling phase remains a key area for improvement to enhance the scalability of the assimilation process.

Summary updated

Abstract

Vision-language models are increasingly deployed as computer-use agents (CUAs) that operate desktops and browsers. Top-performing CUAs are framework-based systems that decompose planning and execution, while end-to-end screenshot-to-action policies are easier to deploy but lag behind on benchmarks such as OSWorld-Verified. GUI datasets like OSWorld pose two bottlenecks: they expose only a few hundred interactive, verifiable tasks and environments, and expert trajectories must be gathered by interacting with these environments, making such data hard to scale. We therefore ask how reinforcement learning from verifiable rewards (RLVR) can best exploit a small pool of exist expert trajectories to train end-to-end policies. Naively mixing these off-policy traces into on-policy RLVR is brittle: even after format conversion, expert trajectories exhibit structural mismatch and distribution shift from the learner. We propose BEPA (Bi-Level Expert-to-Policy Assimilation), which turns static expert traces into policy-aligned guidance via self-rolled reachable trajectories under the base policy (LEVEL-1) and a per-task, dynamically updated cache used in RLVR (LEVEL-2). On OSWorld-Verified, BEPA improves UITARS1.5-7B success from 22.87% to 32.13% and raises a held-out split from 5.74% to 10.30%, with consistent gains on MMBench-GUI and Online-Mind2Web. Our code and data are available at: https://github.com/LEON-gittech/Verl_GUI.git
BibTeX
@misc{wang2026from,
  author = {Zezhou Wang and Ziyun Zhang and Xiaoyi Zhang and Sanglu Lu and Yan Lu},
  title = {{From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation}},
  year = {2026},
  eprint = {2601.05787},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.05787},
}
Figure