Abstract
AIOS 1.0 introduces a paradigm shift in Computer-Use Agents (CUAs) by treating the entire operating system as a structured Model Context Protocol (MCP) server. By transforming raw pixels and complex interfaces into "contextualized environments" that language models can natively understand, the proposed lightweight agent, LiteCUA, achieved a 14.66% success rate on the OSWorld benchmark. This approach addresses the fundamental semantic disconnect between human-centric computer interfaces and LLM reasoning, outperforming several more complex, specialized frameworks.
Introduction
Current AI agents struggle to use computers with the same fluidity as humans. Most research focuses on making the "brain" (the LLM) smarter or the "eyes" (computer vision) sharper. However, a more fundamental hurdle exists: a semantic disconnect.
Computers were designed for human perception—relying on visual cues, spatial layouts, and manual inputs. Forcing an LLM to navigate this "human-first" environment creates immense "interface complexity" that distracts from the actual "decision complexity" of the task. Essentially, the agent spends too much cognitive effort trying to figure out where a button is rather than why it should click it.
Methodology: AIOS 1.0 and LiteCUA
The researchers propose AIOS 1.0, an Agent Operating System designed to bridge this gap through Environmental Contextualization. This system implements the Model Context Protocol (MCP) to abstract the computer's state. Instead of the agent seeing a mess of pixels, it receives well-structured JSON schemas representing interactive elements.
To test this platform, the authors developed LiteCUA, a lightweight agent utilizing an Orchestrator-Worker architecture:
- Perceptor: Combines screenshots with the "Accessibility (A11y) Tree" to build a semantic map (e.g., identifying a blue box as a 'Search' button).
- Reasoner: Interprets the environment against the task goals and world rules.
- Actor: Translates high-level intentions into atomic actions like
CLICK,SCROLL, orTYPEusing thepyautoguilibrary.
Experimental Evaluation
The primary evaluation was conducted on OSWorld, a comprehensive benchmark for multimodal agents consisting of 369 tasks across office productivity, web browsing, and system administration domains.
Key Results
LiteCUA achieved a 14.66% success rate on OSWorld. While the absolute number reflects the difficulty of general computer use, it represents a significant improvement over standalone models and existing frameworks. LiteCUA outperformed GPT-4o (11.21%) and Gemini-1.5-pro (5.1%), as well as specialized agent frameworks like Friday (11.11%) and Open-Interpreter (8.94%).
Discussion and Insights
- Decoupling Complexity: The core insight is that we shouldn't just build better agents; we should build better environments for agents. By decoupling the interface's messiness from the logic of the task, agents can reason more effectively over longer horizons.
- The Semantic Bridge: Using the Accessibility Tree alongside visual screenshots provides a "dual-stream" perception. The A11y tree provides the structural "DNA" of the UI, while the screenshot provides visual context, making grounding much more reliable.
- Limitations: Despite these improvements, general computer use remains a "grand challenge." LiteCUA performed best on structured OS-related tasks (54.2% success) but struggled with complex productivity apps like LibreOffice Calc, which require extremely high-precision perception and long-term planning.
Future work will focus on improving temporal reasoning—understanding how the environment changes over time—and developing more sophisticated semantics for probabilistic reasoning in uncertain UI environments.
Summary updated