Abstract: Many real-world problems are inherently hierarchically structured. The use of this structure in an agent’s policy may well be the key to improved scalability and higher performance. However, standard policy search algorithms often struggle to exploit this structure, frequently falling victim to the "averaging problem" where the learner attempts to average across multiple distinct optima, resulting in poor performance. Hierarchical Relative Entropy Policy Search (HiREPS) addresses this by treating options as latent variables and introducing an entropy-based constraint to maintain diverse, separable solutions.
The Challenge of Multi-Modal Landscapes
In reinforcement learning, especially within robotics, tasks often present multiple optimal solutions. A standard learner might try to average these modes, leading to a policy that performs poorly—for instance, trying to navigate through an obstacle because the best paths lie to its left and right. This "averaging problem" causes existing methods to collapse into a single mode of the solution space, limiting versatility and slowing convergence.
Methodology: Information-Theoretic Hierarchy
HiREPS builds upon Relative Entropy Policy Search (REPS), which regularizes policy updates by bounding the Kullback-Leibler (KL) divergence between successive policies to ensure stable learning. The key technical innovations in HiREPS include:
- Latent Variable Formulation: The choice of a sub-policy or "option" ($o$) is treated as a latent variable. The policy update is reformulated as an Expectation-Maximization (EM)-like process, where the E-step estimates the responsibility of each option for an action, and the M-step updates the options accordingly.
- Entropy Bounding: To prevent options from overlapping and merging, the authors introduce a constraint on the expected entropy of the option distribution. This forces options to remain separable in the action space, ensuring the agent learns truly diverse strategies.
Experimental Evaluation
The paper evaluates HiREPS on three simulated benchmarks: a 2D Two-Gaussians task, a continuous-action PuddleWorld, and a complex robotic Tetherball simulation. In PuddleWorld, HiREPS successfully identified both optimal paths to the goal, whereas standard REPS took significantly longer as it initially tried to average the paths, leading the agent directly into a puddle.
In the Tetherball experiment, where an agent must hit a ball around a pole to reach a target, the solution space is highly multi-modal. HiREPS reached higher average rewards significantly faster than standard REPS. The entropy bound ($\kappa$) was shown to be critical; without it ($\kappa = \infty$), the hierarchy tends to collapse, losing the benefits of the multi-modal representation.
Key Insights and Future Directions
The most significant finding is that hierarchy alone is insufficient for multi-modal tasks; an explicit entropy bound is required to maintain option separability. This diversity makes the agent more robust, as it can maintain up to 12 distinct successful strategies simultaneously.
While HiREPS currently uses options that last for a single time step, future work aims to incorporate temporal abstraction, allowing options to span multiple steps. Additionally, the authors plan to move beyond simulation to validate the algorithm on physical robotic platforms, such as the Barrett WAM arm.
Summary updated