Portable Option Discovery for Automated Learning Transfer in Object-Oriented Markov Decision Processes

Nicholay Topin, Nicholas Haltmeyer, Shawn Squire, John Winder, Marie desJardins, James MacGlashanIJCAI 2015

Introduction

Reinforcement Learning (RL) has seen remarkable success in solving complex tasks, yet most agents remain "one-trick ponies." These agents are typically trained for a single, specific task within a static environment. Even minor modifications to the environment—such as changing the layout of a room or adding new obstacles—often force the agent to restart the learning process from scratch. Portable Option Discovery (POD) is a novel framework designed to overcome this limitation by allowing agents to autonomously discover and transfer high-level skills, or "options," between diverse environments.

The Problem: The "One-Trick Pony" Constraint

To accelerate learning, researchers often employ "options," which are macro-actions like "navigate to the door" or "pick up the passenger." However, traditional option discovery methods assume a consistent state space. When moving from a source domain to a target domain with a different number of objects or a new spatial configuration, the mapping between these worlds usually requires manual intervention by a human expert. POD addresses this bottleneck by leveraging Object-Oriented Markov Decision Processes (OO-MDPs) to automate the discovery and mapping process without human oversight.

The POD Framework: Automating Portability

The Portable Option Discovery framework operates through a three-stage pipeline designed to identify and ground reusable knowledge:

  1. Abstraction: POD identifies the "Greatest Common Generalization" (GCG), which represents the set of object classes shared between the source and target domains. It then constructs an abstract version of the source policy that filters out irrelevant objects.
  2. Scoring & Mapping Search: Transferring knowledge requires mapping objects from the source to the target (e.g., identifying which block in a new room corresponds to a "key" block). POD uses a novel scoring metric to evaluate how well a specific mapping preserves the original policy's logic. To handle the exponential number of possible mappings, it employs a step-wise greedy search to efficiently find high-scoring candidates.
  3. Grounding: Once an optimal mapping is identified, the abstract option is "grounded" into the target domain, enabling the agent to utilize it as a high-level action immediately.
The POD workflow: from source domains to target domain via abstraction and grounding.

Experimental Methodology

The authors evaluated POD using two classic RL environments adapted for object-oriented transfer:

  • Taxi World: An agent must transport passengers in a grid world. The tests involved transferring knowledge between versions of the world with varying numbers of passengers (e.g., from 2 to 7).
  • Block Dude: A complex side-scrolling puzzle game where the agent stacks blocks to reach an exit. This domain is particularly challenging because it requires maintaining precise spatial relationships between multiple objects.

Analysis of Results

The results demonstrate that POD-enabled algorithms—specifically Portable PolicyBlocks (PPB) and Portable Transfer Options (PTOPs)—significantly outperform standard Q-learning and random mapping baselines. By treating the environment as a collection of objects (e.g., "blocks," "agents") rather than raw coordinates, agents can recognize that a stacking skill learned on two blocks is applicable to a room with many more.

Performance comparison in the Block Dude environment, showing POD methods significantly outperforming baselines.

Key findings include:

  • Significant Speedup: Agents using POD achieved higher cumulative rewards much faster than those learning from scratch.
  • Heuristic Success: The automated scoring metric successfully identified useful mappings. In the Block Dude environment, agents using POD's mappings vastly outperformed those using random mappings, which often resulted in useless or counterproductive behavior.
  • Scalability: The framework proved flexible enough to transfer knowledge from large domains to small ones and vice versa.

Limitations and Future Work

While POD represents a significant step forward, several areas for improvement remain. The current implementation relies on tabular state representations, which may not scale to massive or continuous environments without the integration of Value Function Approximation (VFA). Additionally, while the greedy mapping search is efficient, it does not guarantee global optimality. Future research could explore alternative scoring metrics and the application of POD to heterogeneous domains that share fewer object classes.

Summary updated

BibTeX
@inproceedings{topin2015portable,
  author = {Nicholay Topin and Nicholas Haltmeyer and Shawn Squire and John Winder and Marie desJardins and James MacGlashan},
  title = {{Portable Option Discovery for Automated Learning Transfer in Object-Oriented Markov Decision Processes}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure