Affordance Representation and Recognition for Autonomous Agents

Habtom Kahsay Gidey, Niklas Huber, Alexander Lenz, Alois KnollarXiv 2025 · DOI · arXiv

Abstract

Autonomous agents navigating digital environments face significant challenges in processing verbose web data and maintaining interoperability with diverse APIs. This paper proposes a pattern language consisting of DOM Transduction and Hypermedia Affordances Recognition. These patterns allow agents to distill complex Document Object Models (DOM) into actionable Page Affordance Models and dynamically discover service capabilities via standardized descriptors. The resulting architecture enables the construction of robust internal cognitive maps, reducing computational overhead and increasing agent adaptability across the Web of Things.

Introduction

Building autonomous agents capable of navigating the web or managing Internet of Things (IoT) devices presents two primary architectural bottlenecks. First, the Representation Challenge: raw HTML is notoriously verbose, with empirical studies showing that 80–90% of a DOM often consists of non-semantic noise like scripts, styles, and trackers. This bloat exceeds the context limits of Large Language Models (LLMs), increases latency, and degrades reasoning quality.

Second, the Interoperability Challenge: current agents often rely on brittle, hardcoded API integrations. True autonomy requires agents to "discover" capabilities at runtime, recognizing digital affordances much like humans recognize physical ones—for instance, seeing a button as something to click or a field as something to fill.

Overview of the agent perception architecture, from environmental sources to the internal cognitive map.

The Pattern Language for Digital Affordances

The authors adopt a software engineering approach by defining a Pattern Language—a set of reusable solutions to recurring problems in agent perception and action. They propose two specific patterns to bridge the gap between raw digital environments and an agent's internal reasoning.

1. The DOM Transduction Pattern

This pattern serves as a perceptual filter, transforming a messy, raw DOM into a compact Page Affordance Model (PAM). The transduction process typically follows four stages:

  • Cleaning: Stripping out irrelevant tags (e.g., <script>, <style>).
  • Pruning: Using task-aware logic or embeddings to remove content irrelevant to the agent's current goal.
  • Compact Representation: Converting HTML into token-efficient formats like Emmet notation.
  • LLM-as-Transformer: Utilizing a smaller, cost-effective LLM to summarize the DOM before passing it to a more powerful reasoning model.
The DOM Transduction pattern: transforming raw DOM into a Page Affordance Model.

2. Hypermedia Affordances Recognition

Drawing on REST principles (HATEOAS) and the W3C Web of Things (WoT) framework, this pattern enables agents to interact with unknown services. By parsing a Thing Description (TD)—a JSON-LD document describing interaction affordances (what it can do) and protocol bindings (how to talk to it)—an agent can learn to control a new device without prior specific integration code. This allows for a "Hotel Booking" synergy: an agent uses DOM Transduction to navigate a booking site, finds a link to "Smart Room Controls," and then uses Affordance Recognition to dynamically learn how to adjust the thermostat in that specific hotel room.

Key Insights and Qualitative Results

The authors argue that digital agents require "world models" similar to those used by physical robots. These "cognitive maps" should prioritize explicit structure over raw pixel data for efficiency and determinism. By decoupling the agent from specific resources through standardized semantic descriptions, the environment itself guides the agent's interactions.

While this paper presents a conceptual framework, it highlights significant qualitative benefits:

  • Efficiency: Reducing HTML tokens by up to 90% significantly lowers computational costs.
  • Adaptability: Agents can integrate new services at runtime without manual code updates.
  • Scalability: Provides a principled way to build agents that can handle the vast "Web of Things" ecosystem.

Limitations and Future Work

The framework faces challenges such as the risk of "aggressive pruning" deleting necessary metadata (like CSRF tokens) and the requirement for high-quality service descriptions from providers. Future work aims to integrate these structure-based patterns with multi-modal visual perception to create even more resilient autonomous systems that can switch to visual parsing when structured data is missing or obfuscated.

Summary updated

Abstract

The autonomy of software agents is fundamentally dependent on their ability to construct an actionable internal world model from the structured data that defines their digital environment, such as the Document Object Model (DOM) of web pages and the semantic descriptions of web services. However, constructing this world model from raw structured data presents two critical challenges: the verbosity of raw HTML makes it computationally intractable for direct use by foundation models, while the static nature of hardcoded API integrations prevents agents from adapting to evolving services. This paper introduces a pattern language for world modeling from structured data, presenting two complementary architectural patterns. The DOM Transduction Pattern addresses the challenge of web page complexity by distilling} a verbose, raw DOM into a compact, task-relevant representation or world model optimized for an agent's reasoning core. Concurrently, the Hypermedia Affordances Recognition Pattern enables the agent to dynamically enrich its world model by parsing standardized semantic descriptions to discover and integrate the capabilities of unknown web services at runtime. Together, these patterns provide a robust framework for engineering agents that can efficiently construct and maintain an accurate world model, enabling scalable, adaptive, and interoperable automation across the web and its extended resources.
BibTeX
@misc{gidey2025affordance,
  author = {Habtom Kahsay Gidey and Niklas Huber and Alexander Lenz and Alois Knoll},
  title = {{Affordance Representation and Recognition for Autonomous Agents}},
  year = {2025},
  eprint = {2510.24459},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.24459},
}
Figure