Guided Imitation of Task and Motion Planning

Michael James McDonald, Dylan Hadfield-Menell

TL;DR

This paper bridges the gap between modern, data-efficient policy learning (good for short tasks) and traditional Task and Motion Planning (TAMP) (good for long, complex tasks). They propose training a feed-forward policy to imitate the output of a TAMP solver, creating a system that can handle long-horizon manipulation tasks from sensory data. This approach achieves high success rates in complex robotics benchmarks, outperforming prior methods.

What Problem Does This Paper Address?

Robots are getting better at manipulation, but there's a persistent challenge: how to make them perform complex, multi-step tasks that unfold over extended periods. Modern deep learning methods excel at learning policies from sensory input for relatively short tasks, but they struggle with long horizons and intricate sequences of actions. On the other hand, classical Task and Motion Planning (TAMP) methods are designed for these long-horizon, multi-goal problems. However, TAMP is computationally expensive and requires precise knowledge of the robot's state and the environment, making it brittle and difficult to deploy in real-world, noisy conditions. This paper aims to combine the strengths of both worlds: the long-horizon planning capability of TAMP with the robustness and data efficiency of learned policies.

Methods

The core idea is to train a policy to imitate a TAMP solver. This is achieved through a novel hierarchical policy architecture and a distributed TAMP system designed for high throughput.

  • Distributed TAMP Solver: They build an asynchronous, distributed TAMP solver. This system is optimized for generating many TAMP solutions quickly, providing ample supervision data for policy learning. It also incorporates partially trained control policies to speed up the TAMP solver itself, creating a virtuous cycle where learning accelerates planning, and faster planning provides more supervision.
  • Hierarchical Policy Architecture: The learned policy is split into two parts:
  • Task-Level Policy ($\pi_{hi}$): This policy takes observations and a goal and outputs a discrete choice of abstract action and its parameters.
  • Motion-Level Policy ($\pi_{lo}$): This policy takes the abstract action, its parameters, and the current observations to output continuous control commands. It uses an attention module to focus on relevant parts of the observation and then employs action-specific control networks.
  • Policy-Aware Supervision: To combat the issue of small deviations accumulating over time in imitation learning, they employ a "policy-aware supervision" technique. This involves biasing trajectory optimization to provide supervision closer to the states encountered by the learned policies, effectively keeping the supervision data relevant and high-quality.

Datasets

The paper evaluates their approach on several benchmarks:

  • 2D Pick-Place Simulator: A synthetic environment for testing basic pick-and-place tasks.
  • RoboSuite [21]: A benchmark for robotic manipulation tasks, used here for pick-and-place with 7-DoF joint control.
  • RoboDesk [22]: A multitask benchmark featuring 9 diverse tasks for a 7-DoF robotic arm, evaluated using RGB images.

The paper also mentions using their own generated datasets from the TAMP solver and exploration node for training.

Key Insights and Ideas

  • Imitation of TAMP is a powerful paradigm: Instead of trying to make TAMP faster or more robust directly, learning to mimic its output offers a way to achieve its long-horizon capabilities with the benefits of learned policies.
  • Hierarchical policy mirrors TAMP structure: Aligning the learned policy's architecture with the TAMP domain description (task and motion levels) significantly improves learning performance.
  • Distributed TAMP for data generation: Designing the TAMP solver for high throughput, rather than just low latency, is crucial for generating enough supervision data to train complex policies.
  • Partially trained policies accelerate TAMP: Using learned policies to assist the TAMP solver creates a feedback loop that speeds up planning and improves the quality of supervision.
  • Policy-aware supervision is key for stability: Simply imitating TAMP output isn't enough; actively guiding the supervision process based on the learned policy's behavior is essential for robust learning.

Key Results

  • RoboSuite (object pose observations): Achieved 88% success rate on a 4-object pick-place task.
  • RoboDesk (RGB images): Achieved an average of 79% success rate across 9 diverse tasks.
  • Speedup: Partially trained control policies reduced TAMP planning time by a factor of up to 2.6 in robotic manipulation tasks.
  • 2D Pick-Place: Learned policies achieved 83% success on a 3-object task from RGB images and 88% from object pose observations for 5 objects.
  • Generalization: Policies showed some ability to generalize to novel goals and dynamics, though performance varied.

Limitations and Future Work

  • Physical Robot Deployment: The authors acknowledge that deploying their system on physical robots is the next crucial step. They hypothesize that TAMP-derived policies might transfer better than pure RL policies because they don't rely on detailed dynamics models.
  • Parameter Sharing: Exploring parameter sharing for motion policies could save space and potentially speed up training.
  • Sequence Modeling: For tasks involving object permanence and occlusion (like in RoboDesk), sequence modeling techniques might offer significant benefits.
  • Theoretical Guarantees: Investigating theoretical guarantees for the learned policies, potentially by formalizing the problem as optimizing parameters against a mixed discrete-continuous cost function, is a future direction.
  • Improving Task/Motion Policy Learning: Further improvements in learning the task and motion policies themselves, perhaps by leveraging NLP sequence modeling techniques or advanced imitation learning methods like GAIL, are suggested.

Summary updated

Abstract

While modern policy optimization methods can do complex manipulation from sensory data, they struggle on problems with extended time horizons and multiple sub-goals. On the other hand, task and mot...
BibTeX
@inproceedings{mcdonald2021guided,
  author = {Michael James McDonald and Dylan Hadfield-Menell},
  title = {{Guided Imitation of Task and Motion Planning}},
  booktitle = {Conference on Robot Learning},
  year = {2021},
}
Figure