Overcoming Interaction Primitive Constraints in Autonomous Agents

34 concepts32 papers1 proposal

Imagine trying to navigate the modern web or a smartphone app using only a single-button mouse. You can click on links and type text, but you cannot drag a slider to adjust a price filter, pinch to zoom into a map, long-press to reveal a context menu, or swipe fluidly between pages.

This rigid, simplified world is exactly how most state-of-the-art autonomous agents operate today. While large multimodal models have achieved remarkable breakthroughs in visual reasoning and planning, their physical capability—their action space—remains severely bottlenecked. Most digital agents are restricted to a narrow set of basic interaction primitives, such as discrete clicks and text entry. As a result, even the most intelligent models frequently fail at real-world tasks simply because they lack the physical vocabulary to interact with modern user interfaces.

The Gesture Space Coverage Gap

The mismatch between human interaction patterns and agent action spaces creates what researchers call a gesture-space coverage gap. Modern operating systems and web applications are designed for fluid, continuous inputs, yet automated tools are typically built around discrete, atomic actions.

In mobile application testing, this gap directly limits the discovery of critical software flaws. Kevin Moran et al. observe in Auto-completing Bug Reports for Android Applications that automated GUI exploration tools struggle to capture complex user interactions beyond simple clicks, making it impossible to document reproduction steps for bugs triggered by sophisticated gesture sequences. Furthermore, as the authors demonstrate in Automatically Discovering, Reporting and Reproducing Android Application Crashes, failing to account for gestures like swiping prevents testing tools from ever reaching application states hidden within scrollable or paginated components.

This limitation is not unique to software testing; it also severely restricts assistive technologies. Paul Nuyujukian et al. show in Cortical control of a tablet computer by people with paralysis that while intracortical brain-computer interfaces can successfully decode neural activity into standard point-and-click commands, the lack of support for multi-touch gestures like swiping, scrolling, or dragging prevents users from fully leveraging modern mobile operating systems.

Why Basic Primitives Break Complex Workflows

When agents are confined to basic primitives, their performance collapses when facing sophisticated web applications and commercial platforms.

[Agent Reasoner] ---> High-Level Plan: "Filter by price and add to cart"
                           |
                           v
[Restricted Action Space] ---> Limited to: Click(x,y), Type(text)
                           |
                           X  Cannot execute: Drag-and-Drop Slider, Hover, Multi-tab Coordination
                           v
[Dynamic Web Interface]  ---> Task Failure

In web navigation, Hongliang He et al. note in WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models that excluding complex interactions like drag-and-drop severely limits an agent's utility in advanced web applications. This bottleneck becomes glaringly obvious in e-commerce. Yougang Lyu et al. demonstrate in DeepShop: A Benchmark for Deep Research Shopping Agents that while "deep research" agents possess strong reasoning capabilities, they routinely fail at basic UI interactions like manipulating drag-and-drop sliders or navigating multi-level nested menus. Because they cannot utilize these built-in site filters, agents are forced to rely on highly inefficient search methods, causing their success rates to plummet.

Furthermore, human-like browsing requires multi-context coordination that basic action spaces cannot support. Haohao Luo et al. point out in Browsing Like Human: A Multimodal Web Agent with Experiential Fast-and-Slow Thinking that current agents often lack support for advanced browser interactions like zooming, scaling, or managing multiple tabs simultaneously. Similarly, Yuxuan Lu et al. argue in UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design that capturing subtle interaction modalities like hovering, scrolling, or skimming is absolutely necessary to fully replicate how human users navigate digital environments.

This constraint extends far beyond web browsers:

The Illusion of Simplified Tool-Calling

To make agent training more manageable, developers often abstract user interactions into standardized tool-calling formats or high-level API calls. However, this abstraction introduces severe functional and security trade-offs.

Pengzhou Cheng et al. reveal in Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents that simplified tool-calling formats can mask critical security vulnerabilities. They emphasize that evaluating agent safety requires moving away from discrete API calls toward raw, coordinate-based gestures and continuous swipes.

Moreover, relying exclusively on high-level UI manipulation isolates agents from the broader system environment. As Gongjie Zhang observes in MAI-UI, a UI-only approach leads to brittle, error-prone processes and excludes vital functionalities that cannot be achieved through direct screen interaction. For example, Brandon Trabucco et al. demonstrate in Towards Internet-Scale Training For Agents that browser automation tools often restrict agents from performing tasks like file downloads or local application management, preventing them from executing comprehensive workflows that bridge the web and local environments.

Expanding the Action Space

To overcome these constraints, the research community is actively shifting toward expanding agent action spaces through richer data collection, advanced simulation, and granular control frameworks.

The first major hurdle is data: because human demonstration datasets are often mined automatically, they tend to be dominated by simple interactions. To break this bottleneck, Yeda Song et al. introduce a scalable pipeline in Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents to extract complex gestures from video tutorials, ensuring that training data reflects sophisticated mobile interactions. Similarly, Sidong Feng et al. develop Video2Action: Reducing Human Interactions in Action Annotation of App Tutorial Videos to automatically infer multi-touch gestures like pinching and zooming from raw screen recordings, which are traditionally difficult to capture without underlying system logs.

On the environment side, new benchmarks are forcing agents to learn platform-specific procedural knowledge. Christopher Rawles et al. present AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents, which requires agents to master non-obvious interactions like long-pressing to reveal context menus. To help agents discover these paths autonomously, Haolong Yan et al. propose a multi-turn reinforcement learning framework in GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning. This approach allows agents to move away from rigid supervised memorization and actively learn complex gesture-based interactions like dragging and swiping through trial and error.

Ultimately, building truly capable digital assistants requires a symmetrical evolution of mind and body. By expanding the action space from rigid, discrete commands to fluid, human-like interaction primitives, agents can finally navigate the digital world with the same dexterity as the users they are built to serve.

Key Foundational Work

Auto-completing Bug Reports for Android Applications (2017) — contributed 1 concept elements to this direction.
Playing FPS Games with Deep Reinforcement Learning (2017) — contributed 1 concept elements to this direction.
Automatically Discovering, Reporting and Reproducing Android Application Crashes (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_restricted_action_space_937239f7,
  title = {restricted action space},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/restricted-action-space/rdr_937239f780af4a27bfb5b1b9509df590},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_restricted_action_space_937239f7_v1,
  title = {restricted action space},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/restricted-action-space/rdr_937239f780af4a27bfb5b1b9509df590/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake