Overview
Large language models (LLMs) and large multimodal models (LMMs) have shown immense potential in automating complex tasks. However, translating these capabilities into robust "agents" that can navigate dynamic operating systems remains a significant challenge. Most existing agents are specialized for single environments or follow rigid scripts, struggling when apps crash or workflows require cross-application interaction.
To address this, researchers introduce OSCAR (Operating System Control via state-Aware reasoning and Re-planning). OSCAR is a generalist AI agent designed to autonomously control desktop and mobile operating systems by processing screen images and writing executable Python code. By utilizing a state-machine architecture and task-driven re-planning, OSCAR adapts to real-time feedback and recovers from errors, significantly outperforming previous models on complex, multi-step workflows.
The Challenge of Dynamic Environments
The real world is messy. Workflows often require jumping between different software—like extracting data from Excel to build a PowerPoint presentation—while navigating unexpected pop-ups or application delays. Current agents struggle with this because they lack a "grounded" understanding of the interface and cannot easily pivot when a step fails. OSCAR was built to handle any application using standard mouse and keyboard inputs, processing visual screen data to act as a versatile, cross-platform assistant.
Core Innovations
OSCAR introduces several key mechanisms to achieve robust, generalist control:
1. State-Machine Framework
Instead of a linear "step-by-step" approach, OSCAR operates as a dynamic state machine, moving through states such as Observe, Plan, Execute, and Verify.
If the Verify step detects a failure, the agent does not simply give up or restart from scratch. Instead, it transitions back to the Plan state to formulate a workaround. This task-driven re-planning allows it to fix only the specific sub-task that failed, preventing "error propagation" where one small mistake ruins the entire sequence.
2. Dual GUI Grounding
To accurately "see" and interact with the screen, OSCAR employs a two-pronged grounding approach.
It leverages the Accessibility (A11Y) tree (the underlying data used by screen readers) to obtain precise coordinates of buttons and menus. Simultaneously, it uses Set-of-Mark (SoM) prompting to overlay visual bounding boxes on the screen. This dual method proves far more accurate than relying solely on raw pixels or standard Optical Character Recognition (OCR), especially in dense professional software.
3. Code-Centric Action & Two-Level Planning
Rather than predicting raw coordinates (e.g., "click at 500, 300"), OSCAR generates executable Python code using libraries like PyAutoGUI. This enables semantically clear actions—such as mouse.single_click(id=14)—which are easier for the model to reason about and debug. Furthermore, OSCAR utilizes a two-level planning system: it first decomposes a high-level user request into a list of tasks (Level 1), and then generates specific executable actions for each task (Level 2).
Key Results and Performance
The researchers evaluated OSCAR across several rigorous benchmarks, including GAIA (general AI assistants), OSWorld (dynamic Linux desktop environments), and AndroidWorld (mobile device control). OSCAR set new state-of-the-art (SOTA) records across the board:
- GAIA Benchmark: Achieved a 28.7% average success rate, outperforming the previous best (MMAC at 24.0%). On the most difficult "Level 3" tasks, it reached 13.5%, nearly doubling the previous SOTA of 6.1%.
- OSWorld: Reached a 24.5% success rate, significantly higher than competing agents like FRIDAY (18.8%) and UFO (16.5%).
- AndroidWorld: Demonstrated strong cross-platform generalization with a 61.6% success rate on mobile tasks.
- Efficiency: In successful runs, over 80% of OSCAR’s attempts required fewer than three re-planning cycles, whereas competitors often required four or more.
Limitations and Future Directions
Despite its impressive performance, the authors note several areas for future improvement. First, safety safeguards are critical; an autonomous agent with full OS access needs action filters and human-in-the-loop confirmations to prevent accidental data loss. Second, OSCAR currently lacks lifelong learning capabilities—it does not "remember" how to perform a task better the next time. Finally, the verification system focuses on immediate task success and might miss latent side effects, such as background processes crashing, which could impact long-term stability.
Summary updated