Abstract
Autonomous agents navigating digital environments face significant challenges in processing verbose web data and maintaining interoperability with diverse APIs. This paper proposes a pattern language consisting of DOM Transduction and Hypermedia Affordances Recognition. These patterns allow agents to distill complex Document Object Models (DOM) into actionable Page Affordance Models and dynamically discover service capabilities via standardized descriptors. The resulting architecture enables the construction of robust internal cognitive maps, reducing computational overhead and increasing agent adaptability across the Web of Things.
Introduction
Building autonomous agents capable of navigating the web or managing Internet of Things (IoT) devices presents two primary architectural bottlenecks. First, the Representation Challenge: raw HTML is notoriously verbose, with empirical studies showing that 80–90% of a DOM often consists of non-semantic noise like scripts, styles, and trackers. This bloat exceeds the context limits of Large Language Models (LLMs), increases latency, and degrades reasoning quality.
Second, the Interoperability Challenge: current agents often rely on brittle, hardcoded API integrations. True autonomy requires agents to "discover" capabilities at runtime, recognizing digital affordances much like humans recognize physical ones—for instance, seeing a button as something to click or a field as something to fill.
The Pattern Language for Digital Affordances
The authors adopt a software engineering approach by defining a Pattern Language—a set of reusable solutions to recurring problems in agent perception and action. They propose two specific patterns to bridge the gap between raw digital environments and an agent's internal reasoning.
1. The DOM Transduction Pattern
This pattern serves as a perceptual filter, transforming a messy, raw DOM into a compact Page Affordance Model (PAM). The transduction process typically follows four stages:
- Cleaning: Stripping out irrelevant tags (e.g.,
<script>,<style>). - Pruning: Using task-aware logic or embeddings to remove content irrelevant to the agent's current goal.
- Compact Representation: Converting HTML into token-efficient formats like Emmet notation.
- LLM-as-Transformer: Utilizing a smaller, cost-effective LLM to summarize the DOM before passing it to a more powerful reasoning model.
2. Hypermedia Affordances Recognition
Drawing on REST principles (HATEOAS) and the W3C Web of Things (WoT) framework, this pattern enables agents to interact with unknown services. By parsing a Thing Description (TD)—a JSON-LD document describing interaction affordances (what it can do) and protocol bindings (how to talk to it)—an agent can learn to control a new device without prior specific integration code. This allows for a "Hotel Booking" synergy: an agent uses DOM Transduction to navigate a booking site, finds a link to "Smart Room Controls," and then uses Affordance Recognition to dynamically learn how to adjust the thermostat in that specific hotel room.
Key Insights and Qualitative Results
The authors argue that digital agents require "world models" similar to those used by physical robots. These "cognitive maps" should prioritize explicit structure over raw pixel data for efficiency and determinism. By decoupling the agent from specific resources through standardized semantic descriptions, the environment itself guides the agent's interactions.
While this paper presents a conceptual framework, it highlights significant qualitative benefits:
- Efficiency: Reducing HTML tokens by up to 90% significantly lowers computational costs.
- Adaptability: Agents can integrate new services at runtime without manual code updates.
- Scalability: Provides a principled way to build agents that can handle the vast "Web of Things" ecosystem.
Limitations and Future Work
The framework faces challenges such as the risk of "aggressive pruning" deleting necessary metadata (like CSRF tokens) and the requirement for high-quality service descriptions from providers. Future work aims to integrate these structure-based patterns with multi-modal visual perception to create even more resilient autonomous systems that can switch to visual parsing when structured data is missing or obfuscated.
Summary updated