Hyper-Universal Policy Approximation: Learning to Generate Actions from a Single Image using Hypernets

Dimitrios C. Gklezakos, Rishi Jha, Rajesh P. N. RaoarXiv 2022 · OpenReview

Abstract

Inspired by Gibson's notion of object affordances in human vision, we ask the question: how can an agent learn to predict an entire action policy for a novel object or environment given only a single glimpse? To tackle this problem, we introduce the concept of Universal Policy Functions (UPFs) which are state-to-action mappings that generalize not only to new goals but most importantly to novel, unseen environments. Specifically, we consider the problem of efficiently learning such policies for agents with limited computational and communication capacity, constraints that are frequently encountered in edge devices. We propose the Hyper-Universal Policy Approximator (HUPA), a hypernetwork-based model to generate small task- and environment-conditional policy networks from a single image, with good generalization properties. Our results show that HUPAs significantly outperform an embedding-based alternative for generated policies that are size-constrained. Although this work is restricted to a simple map-based navigation task, future work includes applying the principles behind HUPAs to learning more general affordances for objects and environments.

Introduction

In robotics, the concept of "affordance" suggests that an environment or object inherently indicates the actions an agent can take—for instance, a doorway affords "passage." While humans can interpret a map and immediately understand navigation possibilities, artificial intelligence agents typically require extensive reinforcement learning (RL) to master specific environments.

This research addresses two primary challenges in autonomous navigation:

  1. Generalization: Standard RL policies often fail to adapt to novel floor plans without significant retraining.
  2. Resource Constraints: Edge devices, such as small drones or mobile robots, often lack the computational resources to run large, monolithic universal models.

To solve this, we introduce the Universal Policy Function (UPF), which incorporates an environment description directly into the mapping from state and goal to action.

Methodology: Hyper-Universal Policy Approximation (HUPA)

The core innovation of this work is the Hyper-Universal Policy Approximator (HUPA). Unlike standard approaches that use a fixed network, HUPA utilizes a hypernetwork architecture to generate environment-specific policies.

Comparison between the standard Embedding Approach and the proposed HUPA architecture.

As shown in the figure above, we compare two approaches:

  • Embedding Approach: The environment image ($E$) is compressed into a vector and concatenated with state ($s$) and goal ($g$) as input to a fixed primary network ($P$).
  • HUPA: A hypernetwork ($H$) takes the environment image and generates the actual weights ($\theta_P$) for a small, efficient primary network ($P$). This allows the agent's "brain" to be custom-tailored for the specific map it observes.

Experimental Setup

We evaluated HUPA using a novel 2D navigation dataset consisting of "nine-room" grid worlds. By randomly blocking doors, we generated 164 unique map configurations. The agent's task is to navigate from a starting coordinate to a goal using eight discrete actions. Models were trained on a subset of maps and tested on entirely unseen layouts to evaluate zero-shot generalization.

Key Results and Discussion

A critical finding is that modularity via hypernetworks outperforms simple input concatenation. We utilize the Reachability Ratio (RR) as our primary metric, which measures the fraction of states from which an agent can successfully reach the goal.

Test Reachability comparison between HUPA and the Embedding baseline as a function of training set size.

HUPA demonstrates superior efficiency, particularly when the primary network is highly size-constrained (e.g., 16 to 64 neurons). As illustrated in the performance plot, HUPA maintains significantly higher reachability than the embedding baseline across various training set sizes. This suggests that HUPA effectively captures the underlying "physics" of the maps—recognizing walls and doors—rather than simply memorizing layouts.

Conclusion and Future Work

HUPA provides a robust framework for zero-shot policy generation on resource-constrained devices. While currently demonstrated in 2D grid worlds with discrete actions, future work will explore 3D environments, continuous action spaces, and inferring environment context directly from first-person observations. This approach paves the way for agents that can understand and act upon the affordances of novel environments from a single glimpse.

Summary updated

Abstract

Inspired by Gibson's notion of object affordances in human vision, we ask the question: how can an agent learn to predict an entire action policy for a novel object or environment given only a single glimpse? To tackle this problem, we introduce the concept of Universal Policy Functions (UPFs) which are state-to-action mappings that generalize not only to new goals but most importantly to novel, unseen environments. Specifically, we consider the problem of efficiently learning such policies for agents with limited computational and communication capacity, constraints that are frequently encountered in edge devices. We propose the Hyper-Universal Policy Approximator (HUPA), a hypernetwork-based model to generate small task- and environment-conditional policy networks from a single image, with good generalization properties. Our results show that HUPAs significantly outperform an embedding-based alternative for generated policies that are size-constrained. Although this work is restricted to a simple map-based navigation task, future work includes applying the principles behind HUPAs to learning more general affordances for objects and environments.
BibTeX
@misc{gklezakos2022hyper,
  author = {Dimitrios C. Gklezakos and Rishi Jha and Rajesh P. N. Rao},
  title = {{Hyper-Universal Policy Approximation: Learning to Generate Actions from a Single Image using Hypernets}},
  year = {2022},
  eprint = {2207.03593},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.03593},
}
Figure