Modulated Policy Hierarchies

Alexander Pashevich, Danijar Hafner, James Davidson, Rahul Sukthankar, Cordelia SchmidarXiv 2018 · OpenReview

Abstract

Solving tasks with sparse rewards is a main challenge in reinforcement learning. While hierarchical controllers are an intuitive approach to this problem, current methods often require manual reward shaping, alternating training phases, or manually defined sub tasks. We introduce modulated policy hierarchies (MPH), that can learn end-to-end to solve tasks from sparse rewards. To achieve this, we study different modulation signals and exploration for hierarchical controllers. Specifically, we find that communicating via bit-vectors is more efficient than selecting one out of multiple skills, as it enables mixing between them. To facilitate exploration, MPH uses its different time scales for temporally extended intrinsic motivation at each level of the hierarchy. We evaluate MPH on the robotics tasks of pushing and sparse block stacking, where it outperforms recent baselines.

Introduction

Reinforcement learning (RL) has demonstrated impressive capabilities in simulated environments and games. However, its application to robotics is often hindered by the "sparse reward" problem. In tasks like block stacking, a robot may only receive a reward upon successful completion, making it difficult for standard RL agents to discover the necessary sequence of actions through random exploration.

Hierarchical Reinforcement Learning (HRL) addresses this by decomposing complex goals into simpler sub-skills. However, traditional HRL approaches often suffer from instability during joint training or require extensive manual engineering of sub-tasks and rewards. Modulated Policy Hierarchies (MPH) aims to provide a stable, end-to-end framework capable of learning these hierarchies from scratch using only sparse rewards.

Methods: Modulated Policy Hierarchies

MPH introduces three primary innovations to improve the stability and efficiency of hierarchical learning.

Hierarchical Structure and Modulation

The core of MPH is a multi-level policy structure where higher-level policies (managers) modulate the behavior of lower-level policies (workers).

Hierarchical policy structure illustrating the modulation signal between the manager and worker.

A key finding of the research is that the choice of modulation signal significantly impacts performance. While many HRL systems use categorical "one-hot" signals to select a single skill, MPH utilizes bit-vectors. This binary communication allows for "skill mixing," where the worker can interpolate between different behaviors, providing a richer and more flexible interface than discrete selection.

Stable Joint Training with PPO

Jointly training multiple levels of a hierarchy is notoriously difficult because the environment appears non-stationary to the manager as the worker's policy evolves. MPH addresses this by using Proximal Policy Optimization (PPO) at every level. By constraining the magnitude of policy updates, PPO bounds the non-stationarity of the system, allowing all levels to learn simultaneously and stably.

Multi-Level Intrinsic Motivation

To encourage exploration in sparse-reward settings, MPH employs intrinsic motivation—specifically, curiosity based on dynamics prediction error—at every level of the hierarchy. This creates a natural temporal abstraction: the low-level worker is curious about immediate physical interactions, while the high-level manager is curious about reaching novel states over longer time horizons.

Experimental Evaluation

The authors evaluated MPH on two challenging robotics tasks:

  1. FetchPush: A robot arm must push a block to a target location.
  2. Block Stacking: A Kuka arm must reach, grasp, and stack one block on another. This task is particularly difficult due to the long sequence of precise actions required before any reward is granted.
Overview of the FetchPush and Block Stacking tasks used for evaluation.

Results and Discussion

MPH significantly outperformed several state-of-the-art baselines, including Meta Learning Shared Hierarchies (MLSH) and standard Options frameworks.

  • Success Rates: In the stacking task, MPH achieved a 70% average success rate (with a peak of 98%), compared to 54% for Options and 24% for flat PPO. In the pushing task, MPH reached a 71% success rate, outperforming MLSH's 45%.
  • The Role of Curiosity: Ablation studies showed that removing intrinsic motivation from either the manager or the worker led to a significant drop in performance. This confirms that curiosity at all levels of the hierarchy is vital for solving tasks with sparse rewards.

Conclusion

Modulated Policy Hierarchies offer a promising path toward autonomous robotics by enabling the stable, end-to-end learning of skill hierarchies. By leveraging bit-vector modulation and multi-level intrinsic motivation, MPH can solve complex tasks from sparse rewards without manual sub-task definitions. Future research will focus on extending this framework to deeper hierarchies and more complex, long-horizon environments.

Summary updated

Abstract

Solving tasks with sparse rewards is a main challenge in reinforcement learning. While hierarchical controllers are an intuitive approach to this problem, current methods often require manual reward shaping, alternating training phases, or manually defined sub tasks. We introduce modulated policy hierarchies (MPH), that can learn end-to-end to solve tasks from sparse rewards. To achieve this, we study different modulation signals and exploration for hierarchical controllers. Specifically, we find that communicating via bit-vectors is more efficient than selecting one out of multiple skills, as it enables mixing between them. To facilitate exploration, MPH uses its different time scales for temporally extended intrinsic motivation at each level of the hierarchy. We evaluate MPH on the robotics tasks of pushing and sparse block stacking, where it outperforms recent baselines.
BibTeX
@misc{pashevich2018modulated,
  author = {Alexander Pashevich and Danijar Hafner and James Davidson and Rahul Sukthankar and Cordelia Schmid},
  title = {{Modulated Policy Hierarchies}},
  year = {2018},
  eprint = {1812.00025},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.00025},
}
Figure