Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning

Aviv Netanyahu, Tianmin Shu, Joshua B. Tenenbaum, Pulkit Agrawal

Abstract

Teaching robots to perform tasks from a single demonstration is hindered by the ambiguity of a single example. We propose Graph-based Equivalence Mappings (GEM), a framework that represents spatial goals as sparse graphs of relationships and utilizes "equivalence mappings"—such as rotation or scale invariance—to identify task-critical features. GEM employs a two-phase learning process: initial reward estimation followed by active refinement through targeted queries. Our results on the Watch&Move benchmark show that GEM achieves superior generalization and sample efficiency compared to state-of-the-art baselines, reaching near-optimal performance with as few as 30 queries.

Introduction

In one-shot imitation learning, an agent must infer a general task rule from a single demonstration. However, a single state is often ambiguous; for instance, if a robot sees a fork placed to the left of a plate, it remains unclear whether the goal is the specific coordinate, the relative distance, or the directional orientation. Most existing agents either overfit to the demonstration's specific configuration or require extensive datasets to learn the underlying principle.

Ambiguity in one-shot imitation learning: a single demonstration can lead to multiple valid interpretations in a testing environment.

To address this, we introduce GEM, which treats reward learning as a search for the simplest, most invariant explanation of a task. By identifying which spatial relationships are invariant to transformations like rotation or scaling, the agent can effectively generalize its understanding to entirely new environments.

Methods

GEM represents spatial goals using a graph-based structure where nodes correspond to objects and edges denote spatial relationships. The core innovation lies in Equivalence Mappings, which define invariances for these relationships. For example, a "closeness" relationship is typically rotation-invariant, while a "to the left of" relationship is scale-invariant.

The learning process consists of two phases:

  1. Initial Reward Learning: The agent uses a model-based version of Adversarial Inverse Reinforcement Learning (AIRL) to derive a preliminary reward function from the single demonstration.
  2. Active Reward Refinement: The agent iteratively proposes new graph structures and invariance mappings. It generates "discriminative queries"—states that distinguish between competing hypotheses—and presents them to an oracle. This active approach allows the agent to prune incorrect interpretations and converge on the true goal representation with high efficiency.

GEM also incorporates a sparsity prior, favoring simpler graph explanations (fewer edges) as a form of Occam’s Razor, which helps in identifying the most relevant object interactions.

Experimental Evaluation

We evaluated GEM on the Watch&Move benchmark, a 2D physics simulation featuring nine object rearrangement tasks. Agents were trained on a single demonstration and tested in environments with randomized initial object positions. We also validated the learned rewards in VirtualHome, a 3D household simulator, to assess transferability to complex settings.

Performance comparison of GEM against baselines across eight tasks. GEM consistently reaches higher reward levels with fewer queries.

As shown in Figure 6, GEM significantly outperforms strong baselines, including M-AIRL and ReQueST. In human-in-the-loop experiments, GEM reached near-optimal performance within approximately 30 queries. Qualitative analysis confirmed that GEM successfully discovered the correct underlying rules, such as identifying rotation invariance for proximity tasks.

Discussion and Conclusion

GEM demonstrates that combining graph-based representations with active querying and geometric invariances provides a robust path toward generalizable one-shot imitation learning. By "hallucinating" variations of the demonstration through known symmetries, the agent drastically reduces the need for human data.

Future work will focus on expanding GEM beyond pairwise constraints to handle global patterns and integrating natural language to provide a "warm start" for the graph search. Additionally, exploring more complex geometric invariances will be crucial for applying GEM to a broader range of real-world robotic tasks.

Summary updated

BibTeX
@inproceedings{netanyahu2022discovering,
  author = {Aviv Netanyahu and Tianmin Shu and Joshua B. Tenenbaum and Pulkit Agrawal},
  title = {{Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=gV5P3yHp2D},
}
Figure