Most current AI agents interact with computers through a keyhole. While humans fluidly drag files, zoom into maps, and use complex keyboard shortcuts, agents are often restricted to a "click-and-type" paradigm. This narrow action space is the primary bottleneck preventing AI from evolving from simple chatbots into true digital coworkers.
Recent research is focused on expanding this repertoire, moving beyond discrete clicks toward sophisticated, continuous, and multi-modal interactions that allow agents to navigate the digital world as humans do.
The Bottleneck of Restricted Action Spaces
In early web automation, agents relied on structured data like HTML or accessibility trees to understand a page. However, as Hongliang He and colleagues demonstrate with WebVoyager, real-world websites are "messy." Relying on text-based metadata often fails when elements are dynamic or lack proper labels.
The more significant limitation, however, is the action set itself. As noted in OpenWebVoyager, most agents are confined to clicking, typing, and scrolling. This excludes essential professional workflows. For instance, Difei Gao points out in ASSISTGUI that software like Adobe After Effects requires drawing masks and dragging layers—actions that demand high-precision spatial and temporal coordination that standard LLM-based tools simply cannot express.
From Textual Reasoning to Pixel Grounding
To handle these complex actions, agents must stop "thinking" in text and start "reasoning" in pixels. This shift is known as GUI grounding.
Kanzhi Cheng and the team behind SeeClick argue that agents should operate using only screenshots, mimicking human vision. By pre-training models specifically to map natural language instructions to precise (x, y) coordinates, they enable agents to interact with any interface, regardless of whether the underlying code is accessible.
However, simply seeing the pixels isn't enough; the agent must know how to manipulate its own view. Alex Su introduces the Pixel Reasoner framework, which gives models "active vision" tools like zooming and frame selection. This allows the model to resolve ambiguity in visually dense environments—if a button is too small to identify, the agent learns to zoom in before acting.
Bridging the Data Gap with Synthetic Exploration
The hardest part of expanding an agent's action space is finding data. There are few datasets of humans performing complex "drag-and-drop" or "multi-key shortcut" sequences. Researchers are solving this through two main strategies: autonomous exploration and learning from video.
- Autonomous Exploration: Frameworks like AUTO-Explorer use a "Difference Spot" mechanism. After every action, the agent compares the new screen state to the previous one to identify what changed. This allows the agent to autonomously discover how complex UI components (like sliders or carousels) behave without human supervision.
- Inverse Dynamics from Video: Yu Su proposes Watch & Learn, which treats the millions of tutorial videos on the web as a goldmine. By framing the problem as "inverse dynamics"—inferring which keystrokes and mouse movements produced the visual changes in a video—researchers can generate executable training trajectories for professional software workflows.
Scaling to General Computer Control
The ultimate goal of this research is General Computer Control (GCC). This means moving beyond the browser to handle entire operating systems.
Zhixiong Zeng and the UItron team are building foundational agents that work across Mobile and PC environments. These agents don't just click; they are beginning to integrate direct API calls and code execution to bypass interface latency, as suggested by Oriana Riva in UINav.
As we move toward these more capable agents, the evaluation must also evolve. Xiaoran Yang and colleagues introduced IndusGCC, a benchmark for industrial automation. It highlights that in professional settings, the cost of a "missed click" or a "wrong drag" is high. By expanding the action space, researchers aren't just making agents more versatile—they are making them precise enough to handle the high-stakes complexity of the real world.