Hierarchical Relative Entropy Policy Search

Christian Daniel, Gerhard Neumann, Jan PetersAISTATS 2012 · OpenReview

Abstract: Many real-world problems are inherently hierarchically structured. The use of this structure in an agent’s policy may well be the key to improved scalability and higher performance. However, standard policy search algorithms often struggle to exploit this structure, frequently falling victim to the "averaging problem" where the learner attempts to average across multiple distinct optima, resulting in poor performance. Hierarchical Relative Entropy Policy Search (HiREPS) addresses this by treating options as latent variables and introducing an entropy-based constraint to maintain diverse, separable solutions.

The Challenge of Multi-Modal Landscapes

In reinforcement learning, especially within robotics, tasks often present multiple optimal solutions. A standard learner might try to average these modes, leading to a policy that performs poorly—for instance, trying to navigate through an obstacle because the best paths lie to its left and right. This "averaging problem" causes existing methods to collapse into a single mode of the solution space, limiting versatility and slowing convergence.

Methodology: Information-Theoretic Hierarchy

HiREPS builds upon Relative Entropy Policy Search (REPS), which regularizes policy updates by bounding the Kullback-Leibler (KL) divergence between successive policies to ensure stable learning. The key technical innovations in HiREPS include:

  • Latent Variable Formulation: The choice of a sub-policy or "option" ($o$) is treated as a latent variable. The policy update is reformulated as an Expectation-Maximization (EM)-like process, where the E-step estimates the responsibility of each option for an action, and the M-step updates the options accordingly.
  • Entropy Bounding: To prevent options from overlapping and merging, the authors introduce a constraint on the expected entropy of the option distribution. This forces options to remain separable in the action space, ensuring the agent learns truly diverse strategies.
Visualization of mode discovery: Standard REPS (top) collapses to a single mode, while HiREPS with entropy bounding (bottom) successfully captures both optima in a bi-modal task.

Experimental Evaluation

The paper evaluates HiREPS on three simulated benchmarks: a 2D Two-Gaussians task, a continuous-action PuddleWorld, and a complex robotic Tetherball simulation. In PuddleWorld, HiREPS successfully identified both optimal paths to the goal, whereas standard REPS took significantly longer as it initially tried to average the paths, leading the agent directly into a puddle.

In the Tetherball experiment, where an agent must hit a ball around a pole to reach a target, the solution space is highly multi-modal. HiREPS reached higher average rewards significantly faster than standard REPS. The entropy bound ($\kappa$) was shown to be critical; without it ($\kappa = \infty$), the hierarchy tends to collapse, losing the benefits of the multi-modal representation.

Reward comparison on the tetherball task: HiREPS with an appropriate entropy bound (blue) converges faster and to a higher reward than standard REPS (red dashed) or HiREPS without the bound (green).

Key Insights and Future Directions

The most significant finding is that hierarchy alone is insufficient for multi-modal tasks; an explicit entropy bound is required to maintain option separability. This diversity makes the agent more robust, as it can maintain up to 12 distinct successful strategies simultaneously.

While HiREPS currently uses options that last for a single time step, future work aims to incorporate temporal abstraction, allowing options to span multiple steps. Additionally, the authors plan to move beyond simulation to validate the algorithm on physical robotic platforms, such as the Barrett WAM arm.

Summary updated

Abstract

Many real-world problems are inherently hi- erarchically structured. The use of this struc- ture in an agent’s policy may well be the key to improved scalability and higher per- formance. However, ...
BibTeX
@inproceedings{daniel2012hierarchical,
  author = {Christian Daniel and Gerhard Neumann and Jan Peters},
  title = {{Hierarchical Relative Entropy Policy Search}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2012},
  url = {https://openreview.net/forum?id=J4HdrTNN36i},
}
Figure