Hierarchical Reinforcement Learning for Concurrent Discovery of Compound and Composable Policies

Domingo Esteban, Leonel Dario Rozo, Darwin G. Caldwell

TL;DR

This paper introduces a novel hierarchical reinforcement learning (RL) approach that enables robots to learn complex tasks by concurrently discovering both a "compound" policy for the overall task and multiple "composable" sub-task policies. This method leverages off-policy data and a maximum entropy framework to efficiently learn reusable skills that can be executed concurrently, overcoming limitations of sequential sub-task execution and separate policy learning.

What Problem Does This Paper Address?

Reinforcement learning (RL) for complex robotic tasks is notoriously data-hungry and computationally expensive. A common strategy to mitigate this is to decompose complex tasks into simpler, reusable sub-tasks. However, existing methods often learn these sub-task policies (called "composable policies") independently and sequentially, which is inefficient and not suitable for tasks requiring concurrent sub-task execution. This paper addresses the challenge of learning these composable policies and their combination into a single, effective compound policy simultaneously, using shared experience and enabling concurrent execution.

Methods

The core of the proposed method is a two-level hierarchical RL architecture. At the low level, a set of independent Gaussian policies represent the composable sub-tasks. At the high level, a "compound" policy combines these sub-task policies using learned "activation vectors." These activation vectors determine how the outputs of the composable policies are mixed to form the final action for the complex task.

The paper proposes two ways to model this composition:

  1. Convex Combination: The compound policy's action is a weighted sum of actions sampled from the composable policies.
  2. Product of Gaussians: The compound policy's action is derived from a product of the composable policies' distributions, weighted by the activation vectors.

Crucially, the authors introduce an algorithm that learns both the compound policy and the composable policies simultaneously within the same RL process. This is achieved by exploiting off-policy data collected by the compound policy and using a maximum entropy RL objective (specifically, a variant of Soft Actor-Critic) to encourage exploration and facilitate the learning of multiple policies from shared experience.

Datasets

The paper does not introduce new datasets. Instead, it uses simulated robotic environments for evaluation, including:

  • A 2D particle navigation task.
  • A 3-DoF planar manipulator for reaching and pushing tasks.
  • A CENTAURO robot for a more complex task involving balancing and reaching.

These environments are designed to test the ability to decompose tasks and learn concurrent sub-task execution.

Key Insights and Ideas

  • Concurrent Sub-task Learning: The most significant insight is the ability to learn multiple, potentially concurrent, sub-task policies and the overarching task policy simultaneously from a single stream of experience. This drastically improves sample efficiency compared to sequential learning.
  • Reusability of Composable Policies: The learned composable policies demonstrate good performance even when learned unintentionally as part of a larger task, suggesting their potential for reuse in new, related tasks.
  • Maximum Entropy for Exploration: The use of a maximum entropy RL framework is critical. It encourages the compound policy to explore more broadly, generating richer data that benefits the learning of all policies, including the unintentionally learned composable ones.
  • Activation Vectors as a Control Mechanism: The learned activation vectors effectively act as a dynamic controller, allowing the robot to prioritize or blend different sub-task skills as needed for the overall task.

Key Results

The experiments demonstrate that the proposed Hierarchical Intentional-Unintentional (HIU) approach successfully learns to solve complex tasks. The learned compound policies achieve performance comparable to policies trained with single-task RL. Notably, the composable policies learned unintentionally also perform well, sometimes even exceeding the performance of policies trained in isolation. This is attributed to the richer exploration provided by the compound policy. The paper shows that the second composition strategy (product of Gaussians) often converges faster and achieves higher returns.

Limitations and Future Work

  • Temperature Parameter Tuning: The authors acknowledge that choosing the temperature parameters for the maximum entropy objective is challenging and can impact performance. They suggest future work could focus on automating this process.
  • Action Value Function Utilization: Currently, the action value functions of composable policies are only used for evaluation. The paper suggests that these functions could be more actively incorporated into the compound policy's learning process.
  • Task Decomposition Complexity: The current work focuses on tasks decomposable into two subtasks. Future research could explore tasks with more subtasks or scenarios where action vector components are more distinctly assigned to specific tasks (e.g., bimanual tasks).
  • Reusing Composable Policies: Further investigation is needed to understand how the performance of new compound tasks is affected when reusing composable policies learned in different contexts.

Summary updated

Abstract

A common strategy to deal with the expensive reinforcement learning (RL) of complex tasks is to decompose them into a collection of subtasks that are usually simpler to learn as well as reusable for new problems. However, when a robot learns the policies for these subtasks, common approaches treat every policy learning process separately. Therefore, all these individual (composable) policies need to be learned before tackling the learning process of the complex task through policies composition. Moreover, such composition of individual policies is usually performed sequentially, which is not suitable for tasks that require to perform the subtasks concurrently. In this paper, we propose to combine a set of composable Gaussian policies corresponding to these subtasks using a set of activation vectors, resulting in a complex Gaussian policy that is a function of the means and covariances matrices of the composable policies. Moreover, we propose an algorithm for learning both compound and composable policies within the same learning process by exploiting the off-policy data generated from the compound policy. The algorithm is built on a maximum entropy RL approach to favor exploration during the learning process. The results of the experiments show that the experience collected with the compound policy permits not only to solve the complex task but also to obtain useful composable policies that successfully perform in their corresponding subtasks.
BibTeX
@inproceedings{esteban2019hierarchical,
  author = {Domingo Esteban and Leonel Dario Rozo and Darwin G. Caldwell},
  title = {{Hierarchical Reinforcement Learning for Concurrent Discovery of Compound and Composable Policies}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=4KgO9AcZmm-},
}
Figure