Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning

Rashmeet Kaur Nayyar, Siddharth Srivastava

Abstract

Abstraction is key to scaling up reinforcement learning (RL). However, autonomously learning abstract state and action representations to enable transfer and generalization remains a challenging open problem. This paper presents a novel approach for inventing, representing, and utilizing options, which represent temporally extended behaviors, in continual RL settings. Our approach addresses streams of stochastic problems characterized by long horizons, sparse rewards, and unknown transition and reward functions. Our approach continually learns and maintains an interpretable state abstraction, and uses it to invent high-level options with abstract symbolic representations. These options meet three key desiderata: (1) composability for solving tasks effectively with lookahead planning, (2) reusability across problem instances for minimizing the need for relearning, and (3) mutual independence for reducing interference among options. Our main contributions are approaches for continually learning transferable, generalizable options with symbolic representations, and for integrating search techniques with RL to efficiently plan over these learned options to solve new problems. Empirical results demonstrate that the resulting approach effectively learns and transfers abstract knowledge across problem instances, achieving superior sample efficiency compared to state-of-the-art methods.

TL;DR

The CHiRP (Continual Hierarchical Reinforcement Learning and Planning) framework autonomously discovers and represents "options"—temporally extended behaviors—using symbolic abstractions. By identifying significant changes in state saliency, CHiRP creates a library of reusable, composable building blocks that allow agents to solve complex, long-horizon tasks in continual learning settings with far greater sample efficiency than standard deep RL methods.

Introduction

Reinforcement Learning (RL) often hits a wall when faced with "long-horizon" tasks where the goal is many steps away and "sparse rewards" where the agent rarely gets feedback. While humans solve these by breaking tasks into sub-goals (e.g., "find keys," then "start car"), autonomous agents struggle to discover these sub-goals—known as options—on their own.

Existing methods for option discovery usually fall into two traps: they either require dense rewards to learn anything at all, or they produce "black-box" options that cannot be easily reused or used for high-level planning in new environments. In continual RL, where an agent faces a stream of different but related tasks, the inability to transfer and compose learned skills leads to massive "re-learning" costs.

Methods: The CHiRP Framework

The authors introduce CHiRP, which stands for Continual Hierarchical Reinforcement Learning and Planning. The core machinery relies on several integrated components:

  1. Conditional Abstraction Trees (CATs): Instead of looking at the raw state, CHiRP builds a tree-based abstraction that groups states based on their impact on the Q-function (expected reward).
  2. Context-Specific Saliency: CHiRP identifies "context variables"—variables that stay stable for a while but are crucial for the current sub-task (like whether a passenger is currently in a taxi).
  3. Autonomous Option Invention: By monitoring when the state abstraction changes significantly (e.g., the agent moves from a "navigation" context to a "passenger-pickup" context), CHiRP identifies natural endpoints for new options.
  4. Symbolic Signatures: Each invented option is given a symbolic description (initiation and termination sets). This makes the options composable; the agent can use an A* search to "string together" options to solve a brand-new task without starting from scratch.
  5. Continual Learning Loop: As the agent encounters new tasks, it checks its library of options. If a gap exists, it invents a new option signature, learns the policy for it, and adds it to the library.
The CHiRP framework architecture (a) and the process of inventing abstract options using Conditional Abstraction Trees (b).

Experimental Setup

The researchers evaluated CHiRP on five challenging domains, creating significantly larger and more complex versions than those typically found in RL literature:

  • Maze World: Navigation through random wall obstacles.
  • Four Rooms World: Moving between rooms via narrow hallways.
  • Office World: A complex task requiring the collection of coffee and mail before delivery.
  • Taxi World: A hybrid state-space task involving picking up and dropping off passengers at random locations.
  • Minecraft: A resource-gathering and crafting task (mining resources to build tools).

Key Insights and Ideas

  • Abstraction as a Cue for Discovery: The most elegant "aha moment" is using the change in state abstraction as the trigger for option discovery. If the variables the agent needs to pay attention to suddenly shift, it’s a strong signal that one sub-task has ended and another has begun.
  • Symbolic Planning + RL: By representing options symbolically, the authors bridge the gap between high-level classical planning (which is fast but requires models) and low-level RL (which is slow but handles uncertainty).
  • The Three Desiderata: The paper emphasizes that for options to be truly useful in continual learning, they must be composable (chainable), reusable (transferable), and mutually independent (learning one shouldn't break another).

Key Results

CHiRP demonstrated superior performance across all benchmarks:

  • Superior Sample Efficiency: Across all five domains, CHiRP reached high success rates significantly faster than baselines like PPO (Deep RL) and Option-Critic. In many long-horizon cases, PPO failed to solve the tasks entirely within the time budget, while CHiRP succeeded.
  • Zero-Shot Transfer: In the Taxi domain, CHiRP autonomously discovered the four "textbook" options: navigate to passenger, pickup, navigate to destination, and dropoff. Once learned, it could solve new taxi tasks with different start/end points almost instantly by re-planning with these symbols.
  • Interpretability: Unlike neural-network-based options, CHiRP’s options have clear, human-readable definitions based on variable ranges (e.g., "Option 1 terminates when passenger_in_taxi == True").
Performance comparison of CHiRP against baselines across five complex domains, demonstrating superior sample efficiency in terms of fraction of problems solved over time.

Limitations and Future Work

While CHiRP is highly efficient at finding a solution, it doesn't yet guarantee the optimal (shortest) path in all scenarios. Additionally, the system currently requires a predefined set of state variables and a stochastic simulator. Moving toward learning these variables directly from raw pixels (like camera feeds) remains a future challenge. Finally, building and refining CATs adds some computational complexity, though this is offset by the massive reduction in required environment interactions.

Summary updated

Abstract

Abstraction is key to scaling up reinforcement learning (RL). However, autonomously learning abstract state and action representations to enable transfer and generalization remains a challenging open problem. This paper presents a novel approach for inventing, representing, and utilizing options, which represent temporally extended behaviors, in continual RL settings. Our approach addresses streams of stochastic problems characterized by long horizons, sparse rewards, and unknown transition and reward functions. Our approach continually learns and maintains an interpretable state abstraction, and uses it to invent high-level options with abstract symbolic representations. These options meet three key desiderata: (1) composability for solving tasks effectively with lookahead planning, (2) reusability across problem instances for minimizing the need for relearning, and (3) mutual independence for reducing interference among options. Our main contributions are approaches for continually learning transferable, generalizable options with symbolic representations, and for integrating search techniques with RL to efficiently plan over these learned options to solve new problems. Empirical results demonstrate that the resulting approach effectively learns and transfers abstract knowledge across problem instances, achieving superior sample efficiency compared to state-of-the-art methods.
BibTeX
@inproceedings{nayyar2025autonomous,
  author = {Rashmeet Kaur Nayyar and Siddharth Srivastava},
  title = {{Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=pYowb77Fvm},
}
Figure