Composing Diverse Policies for Temporally Extended Tasks

Daniel Angelov, Yordan Hristov, Michael Burke, Subramanian RamamoorthyarXiv 2019 · DOI · arXiv

TL;DR

This paper introduces a framework for sequencing diverse robot control policies—ranging from classical motion planners to deep neural networks—to solve complex, multi-stage tasks. By learning a "Goal Score Estimator" from expert demonstrations, the system can use Model Predictive Control (MPC) to select the best controller for the current situation, achieving a 100% success rate on a challenging physical gear assembly task.

What Problem Does This Paper Address?

Robots often struggle with "temporally extended tasks"—long sequences of actions like assembling a piece of machinery. These tasks usually require different types of skills: reaching for a part needs precise geometric planning, while inserting a gear into a tight slot might require a reactive, vision-based neural network policy.

The core challenge is compositionality: how do you combine these diverse controllers into a single, cohesive system? Most existing methods assume all controllers share the same internal representation (e.g., all are joint-space controllers). In reality, a "reaching" controller might care about joint angles, while an "insertion" controller cares about camera pixels. Manually designing state machines to switch between these is brittle and doesn't scale. The authors aim to automate this sequencing without needing to manually define complex pre- and post-conditions for every sub-task.

Methods

The authors propose a hybrid hierarchical control strategy consisting of three main components:

  1. Controller Library: A collection of pre-existing policies, such as Motion Planning (MP) primitives, Dynamic Motion Primitives (DMPs), and Convolutional Neural Networks (CNNs).
  2. Goal Score Estimator: This is the "progress bar" of the system. It uses a CNN with a Mixture Density Network (MDN) tail—a type of network that predicts a probability distribution rather than a single value—to map image observations to a score between 0 (start) and 1 (goal). It is trained on expert demonstrations to provide a universal metric of task completion.
  3. Forward Dynamics Models: Each controller has its own learned dynamics model that predicts the next latent state (a compressed representation of the world) if that specific controller is executed.
Fig. 2: Demonstrations were performed by using an HTC Vive controller that directly teleoperates the end-effector of the PR2 robot at 20 Hz. In the Gear Assembly Task, our controller library includes motion planning (MP) primitives (operating on joint angles) for picking up or moving and a convolutional neural network (CNN) for inserting the gear (operating on images). The MP primitives produce a trajectory for executing a task. The CNN policy takes a latent representation of the image state and generates a distribution over the target joint angles of the robot. The Forward Dynamics models use a VAE representation alongside an st+1 dynamics prediction network that uses the same decoder. The Goal Score Estimator network takes in an image and produces a distribution over how well this image maps to a particular point in the demonstrations.

To choose a controller, the system uses Model Predictive Control (MPC). It looks $n$ steps into the future for every available controller using their respective dynamics models, evaluates the predicted future state with the Goal Score Estimator, and selects the controller that promises the most progress toward the goal.

Datasets

  • Simulated MDP: A 19-state (and later 100-state) random walk benchmark used to test the framework's robustness to noise in the dynamics models and goal scoring.
  • Gear Assembly Task: A physical experiment on a PR2 robot. The dataset consists of 50 teleoperated demonstrations performed by a human expert using an HTC Vive controller to teach the robot how to pick, move, and insert a gear.

Key Insights and Ideas

  • Universal Currency for Progress: By using images as a shared observation space for the Goal Score Estimator, the framework can compare the "goodness" of vastly different controllers (e.g., one working on joint angles and another on pixels) using a single metric.
  • Planning over Policies: Instead of learning a high-level policy from scratch (which is sample-inefficient), the framework treats the selection of sub-policies as a planning problem. This allows it to leverage the strengths of classical robotics (precision) and modern AI (reactivity).
  • Robustness through Foresight: The MPC look-ahead allows the robot to avoid controllers that might look good in the short term but lead to "dead ends" or regressive states where progress is lost.

Key Results

The framework was evaluated on a complex gear assembly task where the robot had to pick a gear from a table, move it, and insert it onto a peg.

Fig. 1: Robot setup for the gear assembly task. The robot needs to pick up a gear by leveraging the surface of the table, slide it up to an edge, grasp and move it in a collision free manner to the other hand, before inserting the gear onto the base plate.
  • 100% Success Rate: The hybrid "MP & CNN" approach achieved 10/10 successful trials.
  • Failure of Single-Method Approaches: Pure Motion Planning (MP) or DMP approaches failed the insertion phase (1/10 success) because they couldn't handle the small inaccuracies and physical tolerances of the fit. Conversely, pure CNN policies were often too "jerky" for the initial pick-up and violated safety constraints.
  • Noise Tolerance: In simulations, the system remained effective even with significant noise (up to 20% error) in the dynamics and goal scoring models, eventually converging to the optimal sequence of actions.

Limitations and Future Work

While powerful, the method relies on having a library of pre-existing controllers. If a necessary skill is missing from the library, the task cannot be completed. Additionally, the Goal Score Estimator requires expert demonstrations, which may be difficult to obtain for extremely long or dangerous tasks.

Future work could involve automatically discovering new controllers when the current library is insufficient or extending the framework to handle even more diverse sensor modalities beyond vision and joint encoders.

Summary updated

Abstract

Robot control policies for temporally extended and sequenced tasks are often characterized by discontinuous switches between different local dynamics. These change-points are often exploited in hierarchical motion planning to build approximate models and to facilitate the design of local, region-specific controllers. However, it becomes combinatorially challenging to implement such a pipeline for complex temporally extended tasks, especially when the sub-controllers work on different information streams, time scales and action spaces. In this paper, we introduce a method that can compose diverse policies comprising motion planning trajectories, dynamic motion primitives and neural network controllers. We introduce a global goal scoring estimator that uses local, per-motion primitive dynamics models and corresponding activation state-space sets to sequence diverse policies in a locally optimal fashion. We use expert demonstrations to convert what is typically viewed as a gradient-based learning process into a planning process without explicitly specifying pre- and post-conditions. We first illustrate the proposed framework using an MDP benchmark to showcase robustness to action and model dynamics mismatch, and then with a particularly complex physical gear assembly task, solved on a PR2 robot. We show that the proposed approach successfully discovers the optimal sequence of controllers and solves both tasks efficiently.
BibTeX
@misc{angelov2019composing,
  author = {Daniel Angelov and Yordan Hristov and Michael Burke and Subramanian Ramamoorthy},
  title = {{Composing Diverse Policies for Temporally Extended Tasks}},
  year = {2019},
  eprint = {1907.08199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.08199},
}

Loading related content...

Figure