GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

Xiao Zhou, Siyue Zhang, Yilun Zhao, Jinbiao Wei, Tingyu Song, Arman Cohan, Chen ZhaoarXiv 2026 · arXiv

TL;DR

This paper introduces a matched execution-layer benchmark of 440 desktop tasks to fairly compare screen-only GUI agents against skill-mediated CLI agents under identical goals and environments. The study reveals that while out-of-the-box GUI agents outperform CLI agents (59.1% vs 48.2%), the CLI deficit is largely driven by incomplete skill coverage rather than model capability, highlighting distinct execution bottlenecks for each modality.

What Problem Does This Paper Address?

Large Language Model (LLM) agents can control computers either visually (via a Graphical User Interface, or GUI) or programmatically (via a Command Line Interface, or CLI). However, existing benchmarks evaluate these two modalities on completely different tasks, applications, and environments. This confounds the interaction modality (how the agent interacts) with task difficulty or evaluation setup.

Consequently, we don't actually know whether a GUI agent performs better because of visual grounding or because the tasks it is tested on are inherently easier or better specified. This paper aims to isolate the effect of interaction modality by creating a perfectly controlled, head-to-head comparison where goals, initial states, and verifiers are held constant.

Methods

The authors establish a controlled execution-layer protocol that restricts agents to modality-native action spaces:

  • GUI Agents: Restricted to screen-mediated interaction. They receive screenshots of a sandboxed Ubuntu desktop and interact strictly via mouse clicks, drags, typing, scrolling, and keyboard shortcuts.
  • CLI Agents: Restricted to a curated "skill layer" using the CLI-Anything framework. They invoke application-specific programmatic operations rather than executing arbitrary bash/python code or directly editing files.

To dig deeper into the core bottlenecks of each modality, they perform two diagnostic experiments:

  1. Verifier-Guided Skill Patching (CLI): A 4-phase pipeline where missing or broken programmatic skills are repaired using information from the final-state verifier to establish a coverage-controlled upper bound.
  2. Procedure-Guided Grounding (GUI): Rewriting modality-neutral instructions into explicit step-by-step workflow cues to see if providing the exact GUI path mitigates execution errors.

Datasets

The authors introduce a new matched benchmark consisting of 440 desktop tasks spanning 18 real-world applications and 12 workflow categories (such as Visual Design, CAD & 3D, Spreadsheets, and Web). It adapts tasks from OpenComputer and pairs them with programmatic capabilities from CLI-Anything. Crucially, the user goals, initial system states, and executable final-state verifiers are identical across both modalities.

Benchmark composition across workflows and applications

Key Insights and Ideas

  • Modality as a Capability Filter: The choice of interface fundamentally changes how application capability is represented. GUIs embed procedural logic directly into the visual interface (making workflows discoverable), whereas CLIs delegate it to explicit, structured skills.
  • The CLI Deficit is a Coverage Problem: Out-of-the-box CLI agents lag behind GUI agents, but this isn't due to a lack of reasoning or model capability. Original skill libraries only covered 37.6% of verifier checkpoints. When skills are "patched" to match what the application can actually do, CLI performance surges.
  • Complementary Failure Modes: GUI and CLI fail for entirely different reasons. GUI agents struggle with long-horizon visual grounding and state tracking (getting lost in menus or missing confirmation dialogs). CLI agents are heavily bottlenecked by "contract gaps" (missing API functions) and the need to reconstruct implicit defaults (like guessing internal object names that a GUI user inherits for free from the interface).

Key Results

  • Out-of-the-box Performance: The strongest GUI agent (GPT-5.4) achieved a 59.1% full pass rate, outperforming the strongest original-skill CLI agent (Codex GPT-5.5) at 48.2%.
  • The Power of Skill Patching: Augmenting the CLI agent's skill library via verifier-guided patching boosted its success rate to 69.3%, surpassing the best GUI agent.
  • Workflow Preferences: GUI dominated in human-centric, visual interfaces like Audio (73.5% vs 42.9%) and Presentations (78.1% vs 46.9%). Conversely, CLI proved highly competitive or superior in workflows dealing with structured artifacts, like CAD & 3D (67.3% vs 46.9%) and Visual Design.
Figure 4: Distribution of primary error types for failed GUI and CLI tasks.

Limitations and Future Work

  • Strict Modality Isolation: Real-world production agents typically use hybrid approaches (combining screenshots, terminal commands, and direct file edits). This benchmark intentionally disallows hybrid strategies to cleanly isolate modality effects, meaning it does not represent the absolute upper bound of unconstrained agents.
  • Diagnostic Nature of Skill Patching: The patched-skill CLI results represent a counterfactual upper bound because the patching process utilized verifier information. It remains an open question how to automatically construct and scale these skill interfaces with high coverage without relying on verifier-specific data.
  • Overlapping Failure Modes: The failure taxonomy forces each error into a single primary category, though in reality, UI navigation failures and workflow execution failures are often deeply intertwined.

Summary updated

Abstract

Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.
BibTeX
@misc{zhou2026gui,
  author = {Xiao Zhou and Siyue Zhang and Yilun Zhao and Jinbiao Wei and Tingyu Song and Arman Cohan and Chen Zhao},
  title = {{GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents}},
  year = {2026},
  eprint = {2606.24551},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.24551},
}
Figure