Computer-Use Agents Are Embodied Generalists

Pranjal Aggarwal, Sean WelleckarXiv 2026 · arXiv

This paper from researchers at Carnegie Mellon University presents a provocative shift in how we think about robotics and embodied agents. Instead of building specialized models for every physical task, the authors argue that we should treat the physical world as just another piece of software. By wrapping embodied environments in a graphical user interface (GUI), a general-purpose "computer-use agent" (CUA) can navigate, manipulate, and segment objects using the same pixel-based reasoning it uses to navigate a spreadsheet or a web browser.

TL;DR

Embodied AI tasks can be reframed as interacting with digital software. By wrapping physical environments in GUIs, a single general-purpose computer-use agent achieved top rankings in four diverse robotics competitions—spanning navigation and 3D segmentation—without any task-specific tuning.

What Problem Does This Paper Address?

Traditionally, Embodied AI (agents that perceive and act in physical spaces) has relied on specialized architectures. These often include "action heads" for continuous motor control or handcrafted methods for specific tasks like navigation. This fragmentation stands in contrast to fields like NLP, where general-purpose models have replaced specialized ones.

The authors tackle the challenge of generalization: How can we create a single agent capable of performing diverse embodied tasks without redesigning the model every time? They hypothesize that the core skills required for computer use—visual grounding, precise clicking, and exploration—are the exact same skills needed for embodied AI.

Methods

The core innovation is the software wrapper approach. Instead of feeding raw sensor data into a specialized robot controller, the researchers built software applications for each task. These applications:

  1. Render observations of the physical environment onto a screen (screenshots).
  2. Provide controls for acting in that environment via on-screen buttons or keyboard shortcuts.
Figure 1: Computer-use agents can operate any software through keyboard and mouse actions while observing the screen. We argue that physical environments are no different. Once exposed through software, they become just another interface for the same agent.

The agent itself is a "minimal harness" using Gemini 3.5 Flash and Gemini 3.6 Flash. It operates purely through pixels and standard keyboard/mouse actions. Crucially, the agent's design remained fixed across all tasks; only the software interface changed.

Datasets

The authors evaluated their approach by entering four distinct ECCV 2026 workshop competitions, using their single CUA design:

  • VLNVerse: Vision-language navigation in 3D scenes.
  • DialNav: Navigation guided by multi-turn dialogue.
  • OpenSUN3D: Language-guided 3D instance segmentation.
  • MeViSv2: Referring video object segmentation.

They also conducted a preliminary investigation using Gym-Anything for Robotics, comparing a plain vision-language model (Qwen3.5) against a specialized vision-language-action model ($\pi_{0.5}$).

Key Insights and Ideas

  • The GUI as a Universal Abstraction: Representing physical interfaces in software allows researchers to apply the same scaling laws and training methods that improved digital agents to the physical world.
  • Visual Grounding is the Great Equalizer: The paper argues that clicking a tiny "Submit" button on a website is fundamentally the same task as clicking a drawer handle in a 3D scan. Both require precise visual-spatial reasoning.
  • Convergence is Inevitable: Just as computer-use agents moved from needing specialized "Set-of-Mark" overlays to acting on raw pixels, the authors predict Embodied AI will move away from specialized action heads toward general-purpose models that output actions as text tokens.
Figure 3: Parallels between the challenges of computer use and embodied AI.

Key Results

The "zero-tuning" generalist agent performed remarkably well against specialized entries:

  • 1st Place in OpenSUN3D: The agent outperformed all other entries by over 4 points in AP50, demonstrating superior precision in identifying and "clicking" 3D objects.
  • 2nd Place in MeViSv2-Text: It excelled at recognizing when no object matched a description (88.89% accuracy).
  • 3rd Place in DialNav and 6th Place in VLNVerse.
  • GUI vs. API: In a head-to-head comparison, interacting through a GUI was competitive with (and sometimes better than) using a direct text-based API, suggesting that the "pixel-tax" for using a GUI is negligible for modern models.
  • VLM vs. VLA: A standard 9B parameter VLM (Qwen3.5) with no specialized action head matched the performance of a state-of-the-art VLA ($\pi_{0.5}$) in a robot arm simulation, succeeding in 56% of attempts vs. 59%.

Limitations and Future Work

  • Latency: Real-time control remains a hurdle. While the 9B model was relatively fast (400ms per action), it was idle 70% of the time. Robotics requires tighter, faster feedback loops.
  • Complexity Scaling: For highly complex robots (like humanoids with dozens of joints), describing every state and action in plain text might become too slow or token-intensive.
  • Future Directions: The authors advocate for three main research paths:
  • Automating visual grounding data generation using simulators.
  • Building more robotic environments as software-wrapped simulations for RL training.
  • Optimizing architectures for real-time interaction speed.

Summary updated

Abstract

In many areas of AI, general-purpose models have increasingly replaced handcrafted methods and specialized architectures. We argue that embodied AI is poised for a similar shift. Our key claim is that embodied tasks can be formulated as interaction with digital software. Specifically, we build software that renders an agent's view of a physical environment on a screen and provides controls for acting in it. A computer-use agent (CUA) can then perform embodied tasks through the same screenshots and keyboard and mouse actions that it uses to operate ordinary software. Using a single CUA design and changing only the software, our entries win first-, second-, third-, and sixth-place prizes across four ECCV workshop competitions, without any competition-specific agent tuning. These results suggest that the skills CUAs learn on software, such as visual grounding, precise clicking, and exploration, already carry over to embodied tasks, giving CUAs a far broader scope in embodied AI than previously realized. We therefore advocate that computer-use agents can be embodied generalists. Further, we advocate that, by following the training strategy that made computer-use agents general, embodied AI can converge on a single general model as well, ushering in a new era of general embodied AI. Website: https://pranjal2041.github.io/cua-embodied
BibTeX
@misc{aggarwal2026computer,
  author = {Pranjal Aggarwal and Sean Welleck},
  title = {{Computer-Use Agents Are Embodied Generalists}},
  year = {2026},
  eprint = {2611.09847},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09847},
}
Figure