Abstract
Abstraction is key to scaling up reinforcement learning (RL). However, autonomously learning abstract state and action representations to enable transfer and generalization remains a challenging open problem. This paper presents a novel approach for inventing, representing, and utilizing options, which represent temporally extended behaviors, in continual RL settings. Our approach addresses streams of stochastic problems characterized by long horizons, sparse rewards, and unknown transition and reward functions. Our approach continually learns and maintains an interpretable state abstraction, and uses it to invent high-level options with abstract symbolic representations. These options meet three key desiderata: (1) composability for solving tasks effectively with lookahead planning, (2) reusability across problem instances for minimizing the need for relearning, and (3) mutual independence for reducing interference among options. Our main contributions are approaches for continually learning transferable, generalizable options with symbolic representations, and for integrating search techniques with RL to efficiently plan over these learned options to solve new problems. Empirical results demonstrate that the resulting approach effectively learns and transfers abstract knowledge across problem instances, achieving superior sample efficiency compared to state-of-the-art methods.
Introduction
Reinforcement Learning (RL) often suffers from extreme sample inefficiency, particularly in long-horizon environments where rewards are sparse. While hierarchical RL using "options" (temporally extended actions) offers a potential solution, discovering these options autonomously in a way that allows them to be reused across different tasks remains difficult.
CHiRP (Continual Hierarchical Reinforcement Learning and Planning) addresses this by autonomously "inventing" sub-tasks based on symbolic state abstractions. By identifying when the context of a task changes, CHiRP creates a library of reusable behaviors that allow agents to plan their way through complex environments with significantly higher efficiency than standard deep RL methods.
Methods
The CHiRP Framework
The CHiRP architecture integrates three primary components to enable continual learning and planning: the Planner, the CAT+RL module, and the Option Inventor.
- Conditional Abstraction Trees (CATs): Instead of processing raw state data, CHiRP utilizes CATs to create context-specific abstractions. For instance, in a navigation task, the agent may only need high-resolution data for its immediate surroundings until a specific sub-goal is reached, at which point the "context" shifts.
- Autonomous Option Invention: The system monitors CATs for significant shifts in variable saliency (importance for predicting rewards). Such shifts serve as cues for sub-goal completion, triggering the invention of a new option defined by a symbolic signature. This makes the options both interpretable and easy to chain.
- Plannable-CAT: CHiRP maintains a high-level map of its options. When presented with a new task, it uses A* search to plan a sequence of options before execution. If a gap in its library is detected, it creates a "placeholder" option to be learned during the task.
Key Design Principles
- Context as a Cue: Changes in abstraction serve as a proxy for sub-goal completion.
- Symbolic Transfer: Representing options symbolically (e.g., based on variable ranges) allows the agent to immediately assess an option's relevance to a novel task.
- Mutual Independence: The framework seeks options that affect distinct sets of variables, minimizing interference where learning one behavior might degrade another.
Experimental Evaluation
The researchers evaluated CHiRP on five challenging, stochastic domains: Maze World, Four Rooms World, Office World, Taxi World, and a complex Minecraft domain. These environments were designed to test the limits of scalability with continuous or hybrid states and sparse rewards.
As shown in the results, CHiRP consistently outperformed state-of-the-art baselines, including PPO and Option-Critic:
- Sample Efficiency: In the Office and Taxi World domains, CHiRP solved nearly 100% of tasks in the stream, while baselines often failed to solve a single task within the allotted time budget.
- Zero-Shot Planning: By leveraging its learned model of options, CHiRP can often solve new tasks in a stream almost immediately through re-composition, whereas other methods require extensive relearning.
- Interpretability: The discovered options (e.g., "navigate to pickup") align with human intuition, providing a transparent look into the agent's decision-making process.
Conclusion and Future Work
CHiRP demonstrates that integrating symbolic state abstraction with hierarchical planning significantly improves sample efficiency and transferability in continual RL. While the current model assumes full observability and discrete primitive actions, it provides a robust foundation for future work. Upcoming research directions include extending the framework to handle continuous parameterized actions and exploring methods to guarantee the optimality of discovered hierarchies.
Summary updated