OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning

Xiaoqiang Wang, Bang Liu

Overview

Large language models (LLMs) and large multimodal models (LMMs) have shown immense potential in automating complex tasks. However, translating these capabilities into robust "agents" that can navigate dynamic operating systems remains a significant challenge. Most existing agents are specialized for single environments or follow rigid scripts, struggling when apps crash or workflows require cross-application interaction.

To address this, researchers introduce OSCAR (Operating System Control via state-Aware reasoning and Re-planning). OSCAR is a generalist AI agent designed to autonomously control desktop and mobile operating systems by processing screen images and writing executable Python code. By utilizing a state-machine architecture and task-driven re-planning, OSCAR adapts to real-time feedback and recovers from errors, significantly outperforming previous models on complex, multi-step workflows.

The Challenge of Dynamic Environments

The real world is messy. Workflows often require jumping between different software—like extracting data from Excel to build a PowerPoint presentation—while navigating unexpected pop-ups or application delays. Current agents struggle with this because they lack a "grounded" understanding of the interface and cannot easily pivot when a step fails. OSCAR was built to handle any application using standard mouse and keyboard inputs, processing visual screen data to act as a versatile, cross-platform assistant.

Core Innovations

OSCAR introduces several key mechanisms to achieve robust, generalist control:

1. State-Machine Framework

Instead of a linear "step-by-step" approach, OSCAR operates as a dynamic state machine, moving through states such as Observe, Plan, Execute, and Verify.

OSCAR's State-Machine Architecture

If the Verify step detects a failure, the agent does not simply give up or restart from scratch. Instead, it transitions back to the Plan state to formulate a workaround. This task-driven re-planning allows it to fix only the specific sub-task that failed, preventing "error propagation" where one small mistake ruins the entire sequence.

2. Dual GUI Grounding

To accurately "see" and interact with the screen, OSCAR employs a two-pronged grounding approach.

GUI-Grounded Observation Process

It leverages the Accessibility (A11Y) tree (the underlying data used by screen readers) to obtain precise coordinates of buttons and menus. Simultaneously, it uses Set-of-Mark (SoM) prompting to overlay visual bounding boxes on the screen. This dual method proves far more accurate than relying solely on raw pixels or standard Optical Character Recognition (OCR), especially in dense professional software.

3. Code-Centric Action & Two-Level Planning

Rather than predicting raw coordinates (e.g., "click at 500, 300"), OSCAR generates executable Python code using libraries like PyAutoGUI. This enables semantically clear actions—such as mouse.single_click(id=14)—which are easier for the model to reason about and debug. Furthermore, OSCAR utilizes a two-level planning system: it first decomposes a high-level user request into a list of tasks (Level 1), and then generates specific executable actions for each task (Level 2).

Key Results and Performance

The researchers evaluated OSCAR across several rigorous benchmarks, including GAIA (general AI assistants), OSWorld (dynamic Linux desktop environments), and AndroidWorld (mobile device control). OSCAR set new state-of-the-art (SOTA) records across the board:

  • GAIA Benchmark: Achieved a 28.7% average success rate, outperforming the previous best (MMAC at 24.0%). On the most difficult "Level 3" tasks, it reached 13.5%, nearly doubling the previous SOTA of 6.1%.
  • OSWorld: Reached a 24.5% success rate, significantly higher than competing agents like FRIDAY (18.8%) and UFO (16.5%).
  • AndroidWorld: Demonstrated strong cross-platform generalization with a 61.6% success rate on mobile tasks.
  • Efficiency: In successful runs, over 80% of OSCAR’s attempts required fewer than three re-planning cycles, whereas competitors often required four or more.

Limitations and Future Directions

Despite its impressive performance, the authors note several areas for future improvement. First, safety safeguards are critical; an autonomous agent with full OS access needs action filters and human-in-the-loop confirmations to prevent accidental data loss. Second, OSCAR currently lacks lifelong learning capabilities—it does not "remember" how to perform a task better the next time. Finally, the verification system focuses on immediate task success and might miss latent side effects, such as background processes crashing, which could impact long-term stability.

Summary updated

Abstract

Large language models (LLMs) and large multimodal models (LMMs) have shown great potential in automating complex tasks like web browsing and gaming. However, their ability to generalize across diverse applications remains limited, hindering broader utility. To address this challenge, we present OSCAR: Operating System Control via state-Aware reasoning and Re-planning. OSCAR is a generalist agent designed to autonomously navigate and interact with various desktop and mobile applications through standardized controls, such as mouse and keyboard inputs, while processing screen images to fulfill user commands. OSCAR translates human instructions into executable Python code, enabling precise control over graphical user interfaces (GUIs). To enhance stability and adaptability, OSCAR operates as a state machine, equipped with error-handling mechanisms and dynamic task re-planning, allowing it to efficiently adjust to real-time feedback and exceptions. We demonstrate OSCAR’s effectiveness through extensive experiments on diverse benchmarks across desktop and mobile platforms, where it transforms complex workflows into simple natural language commands, significantly boosting user productivity. Our code will be open-source upon publication.
BibTeX
@inproceedings{wang2025oscar,
  author = {Xiaoqiang Wang and Bang Liu},
  title = {{OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=VuTrZzrPfn},
}
Figure