Build the web for agents, not agents for the web

Xing Han Lù, Gaurav Kamath, Marius Mosbach, Siva ReddyarXiv 2025 · DOI · arXiv

Researchers from McGill University and the Mila – Quebec AI Institute are challenging the current trajectory of AI web agent development. Rather than focusing solely on building increasingly sophisticated agents capable of navigating messy, human-centric web interfaces, they argue for a fundamental shift: building a "web for agents."

The Problem: The Human-Agent Mismatch

Currently, we are in an "arms race" to develop agents that mimic human web usage—parsing bloated Document Object Model (DOM) trees or interpreting visual screenshots. This approach suffers from three critical flaws:

  1. Representational Inefficiency: DOM trees are filled with styling "junk," and screenshots can hide vital information. This leads to massive token usage; a single 20-step task using a GPT-4-based agent can cost upwards of $40.
  2. Brittle Interactions: Minor CSS updates or UI changes frequently break agents trained on specific patterns.
  3. Safety and Resource Strain: The lack of standardized "guardrails" makes it difficult to prevent unauthorized actions or data leaks, while bot-blocking measures like CAPTCHAs inadvertently hinder legitimate assistive agents.

The Solution: Agentic Web Interfaces (AWIs)

The authors propose Agentic Web Interfaces (AWIs)—a new interaction paradigm designed specifically for AI agents. AWIs prioritize standardized, efficient, and safe communication over visual rendering. This framework is built upon six guiding principles intended to make the web more accessible to machine intelligence.

Summary of AWI principles, implementation suggestions, and relevant ML disciplines.

Core Principles of AWIs

  • Standardized: Establishing a unified structure for how agents perceive state and execute actions across different platforms.
  • Human-centric: Ensuring humans remain "in the loop" to steer or pause agent trajectories.
  • Safe: Implementing agent-specific Access Control Lists (ACLs), such as requiring biometric verification for final purchases.
  • Optimally-represented: Providing concise state summaries rather than exhaustive, million-token DOM trees.
  • Efficient to host: Minimizing server load through optimized communication.
  • Developer-friendly: Allowing site owners to implement these interfaces without a total infrastructure overhaul.

Key Insights and Innovations

AWIs represent a "middle ground" between narrow, rigid APIs and overly complex human UIs. They provide a machine-readable "map" of a site's capabilities and current state.

A significant distinction is made between AWIs and the Model Context Protocol (MCP). While MCP is a stateless protocol, AWIs are designed to track client-side state, enabling complex, multi-step reasoning without the need to re-send the entire site history for every action.

To address server strain, the authors suggest Agentic Task Queues. These allow agents to process tasks in the background, preventing automated traffic from impacting the real-time experience of human users. Furthermore, by utilizing Unified Higher-Level Actions (e.g., a single goto command instead of multiple low-level clicks), the framework reduces the likelihood of execution errors.

Future Outlook

While the paper establishes a qualitative roadmap rather than quantitative benchmarks, its primary contribution is a call to action. Moving forward, the authors emphasize that success depends on adoption by website owners, incentivized by better resource management and increased accessibility.

Realizing the AWI vision will require collaboration across several ML disciplines, including Human-Computer AI Interaction (HCAI) for user steering, AI Safety for robust access controls, and NLP for advanced state-summarization tools. By shifting the focus from "vision and parsing" to "standardization and interface design," the community can build more robust and reliable autonomous agents.

Summary updated

Abstract

Recent advancements in Large Language Models (LLMs) and multimodal counterparts have spurred significant interest in developing web agents -- AI systems capable of autonomously navigating and completing tasks within web environments. While holding tremendous promise for automating complex web interactions, current approaches face substantial challenges due to the fundamental mismatch between human-designed interfaces and LLM capabilities. Current methods struggle with the inherent complexity of web inputs, whether processing massive DOM trees, relying on screenshots augmented with additional information, or bypassing the user interface entirely through API interactions. This position paper advocates for a paradigm shift in web agent research: rather than forcing web agents to adapt to interfaces designed for humans, we should develop a new interaction paradigm specifically optimized for agentic capabilities. To this end, we introduce the concept of an Agentic Web Interface (AWI), an interface specifically designed for agents to navigate a website. We establish six guiding principles for AWI design, emphasizing safety, efficiency, and standardization, to account for the interests of all primary stakeholders. This reframing aims to overcome fundamental limitations of existing interfaces, paving the way for more efficient, reliable, and transparent web agent design, which will be a collaborative effort involving the broader ML community.
BibTeX
@misc{lu2025build,
  author = {Xing Han Lù and Gaurav Kamath and Marius Mosbach and Siva Reddy},
  title = {{Build the web for agents, not agents for the web}},
  year = {2025},
  eprint = {2506.10953},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.10953},
}

Loading related content...

Figure