LiteCUA: Computer as MCP Server for Computer-Use Agent on AIOS

Kai Mei, Xi Zhu, Hang Gao, Shuhang Lin, Yongfeng ZhangarXiv 2025 · DOI · arXiv

Abstract

AIOS 1.0 introduces a paradigm shift in Computer-Use Agents (CUAs) by treating the entire operating system as a structured Model Context Protocol (MCP) server. By transforming raw pixels and complex interfaces into "contextualized environments" that language models can natively understand, the proposed lightweight agent, LiteCUA, achieved a 14.66% success rate on the OSWorld benchmark. This approach addresses the fundamental semantic disconnect between human-centric computer interfaces and LLM reasoning, outperforming several more complex, specialized frameworks.

Introduction

Current AI agents struggle to use computers with the same fluidity as humans. Most research focuses on making the "brain" (the LLM) smarter or the "eyes" (computer vision) sharper. However, a more fundamental hurdle exists: a semantic disconnect.

Computers were designed for human perception—relying on visual cues, spatial layouts, and manual inputs. Forcing an LLM to navigate this "human-first" environment creates immense "interface complexity" that distracts from the actual "decision complexity" of the task. Essentially, the agent spends too much cognitive effort trying to figure out where a button is rather than why it should click it.

Methodology: AIOS 1.0 and LiteCUA

The researchers propose AIOS 1.0, an Agent Operating System designed to bridge this gap through Environmental Contextualization. This system implements the Model Context Protocol (MCP) to abstract the computer's state. Instead of the agent seeing a mess of pixels, it receives well-structured JSON schemas representing interactive elements.

To test this platform, the authors developed LiteCUA, a lightweight agent utilizing an Orchestrator-Worker architecture:

  • Perceptor: Combines screenshots with the "Accessibility (A11y) Tree" to build a semantic map (e.g., identifying a blue box as a 'Search' button).
  • Reasoner: Interprets the environment against the task goals and world rules.
  • Actor: Translates high-level intentions into atomic actions like CLICK, SCROLL, or TYPE using the pyautogui library.
The LiteCUA architecture featuring Orchestrator, Perceptor, Reasoner, and Actor components.

Experimental Evaluation

The primary evaluation was conducted on OSWorld, a comprehensive benchmark for multimodal agents consisting of 369 tasks across office productivity, web browsing, and system administration domains.

Key Results

LiteCUA achieved a 14.66% success rate on OSWorld. While the absolute number reflects the difficulty of general computer use, it represents a significant improvement over standalone models and existing frameworks. LiteCUA outperformed GPT-4o (11.21%) and Gemini-1.5-pro (5.1%), as well as specialized agent frameworks like Friday (11.11%) and Open-Interpreter (8.94%).

Performance comparison of LiteCUA against state-of-the-art LLMs and specialized agents on the OSWorld benchmark.

Discussion and Insights

  • Decoupling Complexity: The core insight is that we shouldn't just build better agents; we should build better environments for agents. By decoupling the interface's messiness from the logic of the task, agents can reason more effectively over longer horizons.
  • The Semantic Bridge: Using the Accessibility Tree alongside visual screenshots provides a "dual-stream" perception. The A11y tree provides the structural "DNA" of the UI, while the screenshot provides visual context, making grounding much more reliable.
  • Limitations: Despite these improvements, general computer use remains a "grand challenge." LiteCUA performed best on structured OS-related tasks (54.2% success) but struggled with complex productivity apps like LibreOffice Calc, which require extremely high-precision perception and long-term planning.

Future work will focus on improving temporal reasoning—understanding how the environment changes over time—and developing more sophisticated semantics for probabilistic reasoning in uncertain UI environments.

Summary updated

Abstract

We present AIOS 1.0, a novel platform designed to advance computer-use agent (CUA) capabilities through environmental contextualization. While existing approaches primarily focus on building more powerful agent frameworks or enhancing agent models, we identify a fundamental limitation: the semantic disconnect between how language models understand the world and how computer interfaces are structured. AIOS 1.0 addresses this challenge by transforming computers into contextual environments that language models can natively comprehend, implementing a Model Context Protocol (MCP) server architecture to abstract computer states and actions. This approach effectively decouples interface complexity from decision complexity, enabling agents to reason more effectively about computing environments. To demonstrate our platform's effectiveness, we introduce LiteCUA, a lightweight computer-use agent built on AIOS 1.0 that achieves a 14.66% success rate on the OSWorld benchmark, outperforming several specialized agent frameworks despite its simple architecture. Our results suggest that contextualizing computer environments for language models represents a promising direction for developing more capable computer-use agents and advancing toward AI that can interact with digital systems.
BibTeX
@misc{mei2025litecua,
  author = {Kai Mei and Xi Zhu and Hang Gao and Shuhang Lin and Yongfeng Zhang},
  title = {{LiteCUA: Computer as MCP Server for Computer-Use Agent on AIOS}},
  year = {2025},
  eprint = {2505.18829},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.18829},
}
Figure