Learning Options with Interest Functions

Khimya Khetarpal, Doina PrecupAAAI 2019 · OpenReview

Abstract

Learning temporal abstractions which are partial solutions to a task and could be reused for solving other tasks is an ingredient that can help agents to plan and learn efficiently. In this work, we tackle this problem in the options framework. We aim to autonomously learn options which are specialized in different state space regions by proposing a notion of interest functions, which generalizes initiation sets from the options framework for function approximation. We build on the option-critic framework to derive policy gradient theorems for interest functions, leading to a new interest-option-critic architecture.

Introduction

In hierarchical reinforcement learning (HRL), "options" represent high-level behaviors or skills—such as "exiting a room" or "picking up an object"—that consist of sequences of low-level actions. While the development of the Option-Critic architecture allowed these skills to be learned automatically through gradient descent, it introduced a significant challenge: a lack of specialization.

Without constraints on where an option can be used, agents often fall into "degenerate" solutions. These typically manifest in two ways: the Monolith, where a single option is used for the entire task, or Collapse, where options become so short they effectively revert to standard, non-hierarchical actions. This paper introduces Interest Functions to address these issues, providing a mechanism for agents to learn not just how to perform a skill, but where that skill is most applicable.

The Problem: Degeneracy in Option-Critic

Standard hierarchical models assume every option is available in every state. This lack of spatial constraint prevents the emergence of diverse, reusable skills. For a skill to be truly useful, it should be specialized to specific regions of the state space—much like how the skill of "braking" is only relevant when a car is in motion or approaching an obstacle.

In the absence of such specialization, the hierarchical structure provides no benefit over flat reinforcement learning. The authors argue that by constraining the "interest" of an option, we can force the agent to develop a diverse library of specialized skills rather than a single "jack-of-all-trades" policy.

Methods: The Interest-Option-Critic Architecture

The core innovation of this work is the Interest Function ($I_\omega$), a differentiable function that indicates the degree to which an option is "interested" in being active in a particular state.

From Initiation Sets to Interest Functions

This concept is a modern generalization of the "initiation sets" proposed in the original options framework (1999). While original initiation sets were binary and difficult to optimize, interest functions are parameterized by a vector $z$. This allows the agent to use gradient descent to learn the boundaries of a skill's applicability end-to-end.

The Gradient Theorems

The authors derive three new policy gradient theorems to update the components of the architecture simultaneously:

  1. Interest Functions ($z$): Determines where the skill is applicable.
  2. Intra-option Policies ($\theta$): Determines the actions taken within the skill.
  3. Termination Functions ($\nu$): Determines when the skill should stop.

These updates are integrated into Algorithm 1 (Interest-Option-Critic), which extends the tabular Q-learning framework to include learned interest boundaries.

Key Insights

  • Attention-Based Specialization: By treating the initiation of a skill as an "attention" mechanism, the agent naturally learns to partition the state space into regions of expertise.
  • Mathematical Weighting: The paper proves that introducing interest functions changes the "discounted weighting" of state-option pairs. Gradient updates for a specific skill are now weighted by the interest function, meaning the model only optimizes a skill for the states where it is actually "interested" in operating.
  • End-to-End Learning: Unlike previous attempts to manually define where skills should start, the Interest-Option-Critic allows the agent to discover these boundaries autonomously through interaction with the environment.

Conclusion and Future Work

The primary contribution of this work is theoretical: the derivation of the Interest Function Gradient Theorem. This provides a formal mathematical path toward more specialized and reusable hierarchical agents.

While the current results focus on the theoretical framework and tabular settings, the authors identify the robotics domain as the primary target for future application. The goal is to demonstrate how specialized skills can facilitate lifelong learning in high-dimensional, real-world environments. Future research will focus on scaling these interest functions using deep function approximation (Deep RL) and validating the architecture in complex simulated environments like MuJoCo or Atari.

Summary updated

Abstract

Learning temporal abstractions which are partial solutions to a task and could be reused for solving other tasks is an ingredient that can help agents to plan and learn efficiently. In this work, we tackle this problem in the options framework. We aim to autonomously learn options which are specialized in different state space regions by proposing a notion of interest functions, which generalizes initiation sets from the options framework for function approximation. We build on the option-critic framework to derive policy gradient theorems for interest functions, leading to a new interest-option-critic architecture.
BibTeX
@inproceedings{khetarpal2019learning,
  author = {Khimya Khetarpal and Doina Precup},
  title = {{Learning Options with Interest Functions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=Xziwqdglqjo},
}
Figure