Step-level Optimization for Efficient Computer-use Agents

Jinbiao Wei, Kangqi Ni, Yilun Zhao, Guo Gan, Arman CohanarXiv 2026 · arXiv

TL;DR

State-of-the-art computer-use agents are incredibly expensive and slow because they uniformly invoke massive frontier models at every single interaction step. This paper introduces an event-driven, step-level cascade framework that runs a small, cheap model by default and selectively escalates to a large model only when lightweight monitors detect progress stalls or critical milestones. This approach slashes inference costs by up to 74.6% and latency by 45.8% while retaining nearly all the accuracy of an always-large agent.

What Problem Does This Paper Address?

Computer-use agents interact directly with graphical user interfaces (GUIs) to automate workflows across diverse applications. However, current high-performing agents suffer from a major deployment bottleneck: they allocate compute uniformly, calling expensive frontier models (like Claude 3.5 Sonnet or GPT-4) at every single step. This leads to high latency and per-task costs that can exceed $1.00, making them impractical for real-world production.

The authors argue that uniform compute allocation is fundamentally inefficient because GUI trajectories are highly heterogeneous. Most steps are routine and easily handled by smaller, cheaper models. Failures are not uniformly distributed; instead, they concentrate in a small number of high-risk moments, primarily taking two forms:

  1. Progress stalls: The agent gets stuck in a loop, repeating ineffective actions or failing to make meaningful change.
  2. Silent semantic drift: The agent takes locally plausible actions but has drifted away from the user's actual goal, eventually failing while confidently declaring it is "done."

Methods

The authors propose a plug-and-play, event-driven cascade system. Instead of a static query-level router, it operates dynamically at the step level using two lightweight, text-based monitors trained on short windows of recent interaction history:

  • Stuck Monitor: A fine-tuned ModernBERT model that analyzes a rolling window of recent textual rationales and actions. If it detects a progress stall (e.g., action loops or oscillations), it triggers immediate escalation to a larger model for recovery.
  • Milestone Monitor: Also based on ModernBERT, this monitor additionally conditions on the task description to identify when a semantically meaningful checkpoint is reached. When a milestone fires, it triggers a sparse verification call to a large model verifier to check for semantic drift.

If verification passes, control returns to the small model; if it fails, the large model takes over execution. Because these monitors only process text logs rather than heavy raw screenshots or DOM deltas, they are incredibly cheap to run at every step.

Figure 1: Overview of the proposed event-driven, step-level cascade for computer-use agents. A small policy acts by default while two lightweight monitors analyze the trajectory: a Stuck Event detector triggers escalation when recent steps show repeated actions or lack of progress, and a Milestone Event detector selects semantically meaningful checkpoints for sparse verification by a stronger model, which returns control to the small policy if the check passes and otherwise hands control to the large policy.

Datasets

The framework was evaluated on two prominent multi-step GUI benchmarks:

  • OSWorld: A desktop environment benchmark (Ubuntu VM) covering office tools, browsers, and system utilities.
  • WebArena (Verified): A web-based navigation benchmark spanning multiple realistic online domains.

To train the monitors, the authors collected trajectories from small agents across 300 randomly generated GUI tasks and used GPT-5.2 to provide multi-run consensus labels for stuck states and milestones.

Key Insights and Ideas

  • Quantifiable Failure Signatures: Failed trajectories are significantly longer (2.5× to 2.8×) and exhibit much higher action repetition rates than successful ones, validating the need for a targeted stuck detector.
  • The Ceiling vs. Floor Dynamics: The choice of the large model determines the overall success rate ceiling of the cascade, while the small model primarily dictates the cost floor by handling routine steps.
  • Synergy of Dual Monitors: Ablation studies revealed that combining both the Stuck and Milestone monitors provides a massive performance boost compared to using either alone. They cover entirely different failure modes (local loops vs. global goal drift).
  • Event-Driven Beats Fixed-Interval: Triggering verification based on learned semantic events is far more cost-effective and accurate than checking every fixed $k$ steps, especially in short-horizon web tasks where fixed checks easily miss critical windows.
Quantitative failure signatures of small-policy trajectories

Key Results

  • Massive Cost and Latency Savings: On OSWorld, pairing EvoCUA-8B with Kimi K2.5 achieved a 58.2% success rate (nearly matching standalone Kimi's 60.1%) while reducing monetary cost by 61.4%.
  • Across benchmarks, the cascade reduced inference costs by up to 74.6% and wall-clock latency by up to 45.8%.
  • The lightweight monitors achieved high accuracy on held-out data: the Stuck Monitor reached 93.9% accuracy (91.5% F1), proving that compact textual context is highly predictive of agent behavior.

Limitations and Future Work

To maintain ultra-low latency and cost, the always-on monitors are intentionally restricted to textual reasoning-action traces rather than processing raw screenshots or DOM deltas. While highly effective, this design choice means the monitors might miss purely visual failure modes (e.g., a page failing to load visually while the text logs look normal). The authors suggest that future work could explore incorporating lightweight multimodal features or adaptive thresholding to further optimize the cascade.

Summary updated

Abstract

Computer-use agents provide a promising path toward general software automation because they can interact directly with arbitrary graphical user interfaces instead of relying on brittle, application-specific integrations. Despite recent advances in benchmark performance, strong computer-use agents remain expensive and slow in practice, since most systems invoke large multimodal models at nearly every interaction step. We argue that this uniform allocation of compute is fundamentally inefficient for long-horizon GUI tasks. Such trajectories are highly heterogeneous: many steps are routine and can be handled reliably by smaller, cheaper policies, while errors tend to concentrate at a relatively small number of high-risk moments. Across computer-use benchmarks, these failures repeatedly take two forms: progress stalls, where the agent loops, repeats ineffective actions, or fails to make meaningful progress, and silent semantic drift, where the agent continues taking locally plausible actions after already deviating from the user's true goal. To address this inefficiency, we propose an event-driven, step-level cascade for computer-use agents that runs a small policy by default and escalates to a stronger model only when lightweight learned monitors detect elevated risk. Our framework combines two complementary signals: a Stuck Monitor that detects degraded progress from recent reasoning-action history and triggers recovery, and a Milestone Monitor that identifies semantically meaningful checkpoints where sparse verification is most informative for catching drift. This design turns always-on frontier-model inference into adaptive, on-demand compute allocation over the course of an evolving interaction. The framework is modular and deployment-oriented: it can be layered on top of existing computer-use agents without changing the underlying agent architecture or retraining the large model.
BibTeX
@misc{wei2026step,
  author = {Jinbiao Wei and Kangqi Ni and Yilun Zhao and Guo Gan and Arman Cohan},
  title = {{Step-level Optimization for Efficient Computer-use Agents}},
  year = {2026},
  eprint = {2604.27151},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.27151},
}

Loading related content...

Figure