OpenComputer: Verifiable Software Worlds for Computer-Use Agents

Jinbiao Wei, Qianran Ma, Yilun Zhao, Xiao Zhou, Kangqi Ni, Guo Gan, Arman CohanarXiv 2026 · arXiv

TL;DR

OpenComputer introduces a fully automated framework to generate realistic desktop tasks and evaluate computer-use agents using robust, hard-coded application verifiers instead of unreliable LLM judges. Spanning 33 applications and 1,000 tasks, it reveals that while frontier models make partial progress, a massive performance gap remains for robust end-to-end computer automation.

What Problem Does This Paper Address?

Building and evaluating "computer-use" agents—AI models designed to control a PC via GUI or CLI actions—faces two massive bottlenecks: environment scaling and trustworthy verification.

Manually creating realistic initial states (populating files, setting up databases, configuring browser histories) is slow, tedious, and expensive. Worse, evaluating whether an agent actually succeeded usually relies on an "LLM-as-a-judge" analyzing screenshots. This approach frequently fails because critical errors can be visually imperceptible (e.g., typing data into a single spreadsheet cell instead of two adjacent ones) or hidden entirely from view (e.g., background terminal logs, file metadata, or database schemas).

Methods

OpenComputer shifts verification from a downstream afterthought to the core organizing principle of environment synthesis. It operates in four tightly coupled phases:

  1. Verifier Generation: Automatically builds application-specific Python modules that query low-level, stable inspection channels (like browser debugging protocols, D-Bus, LibreOffice UNO, or SQLite databases) to output structured JSON states.
  2. Self-Evolving Verification: To fix brittle assumptions or schema mismatches, a "self-evolution loop" runs a strong agent on calibration tasks. It flags disagreements between the programmatic verifier and a high-level LLM judge, automatically refining the verifier's code or documentation based on execution feedback.
  3. Task Generation Pipeline: Proposes realistic user goals, filters them for complexity, matches them against inspectable verifier endpoints, and automatically synthesizes the required environment files, folders, and configurations.
  4. Evaluation Harness: Executes the agent in a fresh sandbox, records the trajectory, and runs the hard-coded verifier commands to compute precise, partial-credit rewards based on actual system state.
Figure 1: Overview of the OpenComputer verifiable software-world synthesis pipeline. Phase 1 generates app- specific verifier endpoints over the most reliable inspection channels and validates it with unit and integration tests for structured, machine-checkable state. Phase 2 closes a self-evolving loop: calibration tasks drive a strong agent run, an LLM evaluator and the programmatic verifier produce verdicts that disagreement analysis attributes, and verifier memory + checker/endpoint/doc fixes refine the verifier with execution-grounded feedback. Phase 3 proposes user goals, filters by complexity and data generatability, matches against the verifier, synthesizes the environment, and emits a final task instance. Phase 4 runs the agent and computes the reward.

Datasets

The authors introduce the OpenComputer Benchmark, a large-scale suite containing:

  • 33 desktop applications covering browsers, office tools, creative software, development environments, and communication tools.
  • 1,000 finalized, machine-checkable tasks.
  • An average of 17.7 verifier endpoints per app and 6.9 checks per task, enabling fine-grained partial credit.

Unlike existing benchmarks like OSWorld, which heavily rely on manual curation or visual proxies, OpenComputer's tasks and environment initializations are entirely automatically generated and grounded in deep state verification.

Figure 2: Example application endpoint specification used by OpenComputer verifiers.

Key Insights and Ideas

  • Pixels Lie, State Doesn't: Hard-coded verifiers align significantly better with human adjudication (94.1% task-level alignment) than an agentic LLM-as-a-judge (79.2%). LLM judges are easily fooled by "near-miss" visual outputs where an agent makes a tiny but fatal error (e.g., formatting the wrong cell or missing a terminal error log).
  • The Cross-Benchmark Illusion: Open-source models that perform decently on existing benchmarks collapse on OpenComputer. This reveals that current open-source agents suffer from severe overfitting to specific benchmark styles and fail to generalize to broader, heterogeneous software environments.
  • Visual Grounding is Essential Even for CLI: When comparing GUI agents to CLI agents (using tools like Claude Code) on a shared subset of tasks, GUI agents still achieved higher pass rates. This indicates that visual interaction provides crucial context and grounding that text-based terminal interfaces lack, even for technically solvable tasks.

Key Results

  • Frontier Models Lead but Struggle: GPT-5.4 achieved the highest full-task success rate at 68.3% (with an 88.4% average partial reward), followed by Claude-Sonnet-4.6 (64.4%) and Kimi-K2.6 (58.8%). Even the best models fail to completely solve nearly a third of the tasks.
  • Open-Source Collapse: Open-source models showed dramatic performance drops compared to their reported OSWorld scores. For instance, GUI-OWL-1.5-8B dropped from a 52.3% OSWorld score to just 5.7% success on OpenComputer; EvoCUA-8B dropped from 46.1% to 10.9%.
  • Efficiency: GPT-5.4 proved to be the most efficient agent, completing tasks in an average of 19.0 steps, compared to 31.5 steps for Claude-Sonnet-4.6 and 73.6 steps for GUI-OWL.

Limitations and Future Work

  • The Limits of Programmatic State: Not all desktop tasks can be reduced to clean database or file checks. Tasks involving visual or geometric layout judgments—such as verifying if an arrow in Draw.io correctly connects two specific boxes—remain difficult to check purely via application state APIs.
  • Hybrid Verification: The authors identified 17 tasks that required visual/geometric validation and excluded them from the official benchmark to maintain reproducibility. Future work will focus on hybrid verification pipelines that safely combine hard-coded state checks with vision-based LLM judgments.

Summary updated

Abstract

We present OpenComputer, a verifier-grounded framework for constructing verifiable software worlds for computer-use agents. OpenComputer integrates four components: (1) app-specific state verifiers that expose structured inspection endpoints over real applications, (2) a self-evolving verification layer that improves verifier reliability using execution-grounded feedback, (3) a task-generation pipeline that synthesizes realistic and machine-checkable desktop tasks, and (4) an evaluation harness that records full trajectories and computes auditable partial-credit rewards. In its current form, OpenComputer covers 33 desktop applications and 1,000 finalized tasks spanning browsers, office tools, creative software, development environments, file managers, and communication applications. Experiments show that OpenComputer's hard-coded verifiers align more closely with human adjudication than LLM-as-judge evaluation, especially when success depends on fine-grained application state. Frontier agents struggle with end-to-end completion despite partial progress, and open-source models exhibit sharp drops from their OSWorld-Verified scores, exposing a persistent gap in robust computer automation.
BibTeX
@misc{wei2026opencomputer,
  author = {Jinbiao Wei and Qianran Ma and Yilun Zhao and Xiao Zhou and Kangqi Ni and Guo Gan and Arman Cohan},
  title = {{OpenComputer: Verifiable Software Worlds for Computer-Use Agents}},
  year = {2026},
  eprint = {2605.19769},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.19769},
}

Loading related content...

Figure