Hierarchical Reinforcement Learning By Discovering Intrinsic Options

Jesse Zhang, Haonan Yu, Wei Xu

TL;DR

This paper introduces HIDIO, a novel hierarchical reinforcement learning (hRL) method that learns task-agnostic "options" (sequences of actions) in a self-supervised manner. These learned options are then utilized by a higher-level scheduler to solve sparse-reward tasks more efficiently than existing methods.

What Problem Does This Paper Address?

Traditional reinforcement learning (RL) struggles with sparse reward tasks, where agents only receive a reward signal infrequently. Hierarchical RL (hRL) offers a solution by decomposing tasks into subtasks, but existing hRL methods often require manual design of task structures or pre-defined skills. This paper addresses the challenge of automatically learning useful task decompositions and skills (options) without prior knowledge of the task structure, enabling more efficient exploration and learning in complex environments.

Methods

HIDIO employs a two-level hierarchy: a scheduler and a worker.

  • Scheduler: This higher-level policy learns to select options to maximize the overall task reward. It operates on a longer timescale, choosing an option every $K$ environment steps.
  • Worker: This lower-level policy learns to execute primitive actions conditioned on the selected option. Crucially, the worker is trained using a self-supervised intrinsic reward signal. This signal is derived from a discriminator that tries to distinguish between different option sub-trajectories. The worker's objective is to maximize the entropy of the option conditioned on the sub-trajectory (to make options identifiable) while also maximizing its own policy entropy (to encourage diversity). This intrinsic reward allows the worker to learn useful skills even when the environment provides no external reward.

The novelty lies in the worker's self-supervised learning objective, which encourages the discovery of diverse, task-agnostic options that can be composed by the scheduler. The paper also explores various "feature extractors" for the discriminator to better capture relevant information from option sub-trajectories.

Datasets

The paper evaluates HIDIO on two suites of environments:

  1. 7-DOF environments:

    • 7-DOF Reacher
    • 7-DOF Pusher These are standard robotic manipulation tasks with sparse rewards.
  2. SOCIALROBOT environments:

    • Goaltask
    • Kickball These are newly proposed sparse-reward robotic navigation and manipulation tasks.

Key Insights and Ideas

  • Self-supervised option discovery is key: By decoupling option learning from the sparse task reward, HIDIO can learn useful skills even in challenging environments where traditional RL or even other hRL methods fail.
  • Task-agnostic options are generalizable: The learned options are not tied to a specific task, meaning they can potentially be reused or fine-tuned for new tasks, reducing the need for extensive retraining.
  • Discriminator design matters: The choice of features fed into the discriminator significantly impacts the quality of discovered options and overall performance. Features that capture state-action dynamics and transitions proved more effective.
  • Short-term options are sufficient: The paper demonstrates that options with short durations ($K=3$ or $K=5$ steps) are effective, providing temporal abstraction without requiring overly long, complex sub-policies.

Key Results

HIDIO consistently outperforms standard SAC, SAC+ActRepeat, HIRO, and HiPPO across all tested environments, particularly in sparse-reward settings.

  • Robotic Manipulation (Pusher, Reacher): HIDIO achieves near 100% success rates in Pusher, significantly outperforming HiPPO. It also shows strong performance in Reacher, a difficult exploration task.
  • Robotic Navigation (Goaltask, Kickball): HIDIO achieves 60-80% success in Goaltask, where other methods struggle. In Kickball, the most challenging task, HIDIO achieves 30-40% success, while other methods perform poorly.
  • Sample Efficiency: HIDIO demonstrates superior sample efficiency, reaching higher success rates with fewer training steps compared to baselines.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper notes that HIDIO's hyperparameters were mostly heuristically chosen due to the large search space, suggesting that more systematic tuning could further improve performance.
  • Image-based Observations: While the method is designed for non-image observations, the authors suggest that it can be applied to image-based environments by replacing MLPs with convolutional layers, leaving this as future work.
  • Option Length Trade-off: The paper explores option length ($K$) and finds a trade-off between consistency and coverage. Longer options lead to more consistent trajectories but potentially less coverage, while shorter options require more distinct sub-trajectories for the discriminator. The optimal $K$ might be task-dependent.

Summary updated

BibTeX
@inproceedings{zhang2021hierarchical,
  author = {Jesse Zhang and Haonan Yu and Wei Xu},
  title = {{Hierarchical Reinforcement Learning By Discovering Intrinsic Options}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=r-gPPHEjpmw},
}
Figure