Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation

Jacky Liang, Mohit Sharma, Alex LaGrassa, Shivam Vats, Saumya Saxena, Oliver Kroemer

TL;DR

This paper introduces a search-based task planning framework that allows robots to continuously learn and integrate new skills over their lifetime. By training Graph Neural Network-based "Skill Effect Models" (SEMs) to predict the macro-outcomes of parameterized skills, the system can efficiently plan for novel tasks without relying on rigid plan skeletons or predefined subgoals.

What Problem Does This Paper Address?

For a robot to be truly useful in the real world, it needs to acquire new skills over time (lifelong learning) and figure out how to combine them to solve various tasks. Traditional task planning struggles with this because it often relies on rigid assumptions: skills must reach specific "subgoals," share the same underlying implementation, or rely on task-specific plan skeletons.

If a robot learns a completely new type of skill—say, using a tray to sweep multiple objects instead of picking them up one by one—existing planners cannot easily incorporate it without extensive manual re-engineering or fine-tuning. The authors tackle the challenge of building a flexible task planner that can dynamically search through an ever-growing library of diverse, parameterized skills (with continuous, discrete, or mixed parameters) to solve new tasks on the fly.

Methods

The core of the approach is combining Search-Based Task Planning (specifically, Weighted A) with learned Skill Effect Models (SEMs)*.

  • SEMs as Graph Neural Networks (GNNs): Instead of simulating every low-level action, the authors train GNNs to predict the macro-outcome of a skill. The GNN treats objects in the scene as nodes. Given a current state and a set of skill parameters, the SEM predicts the final state of the objects and the execution cost of the skill.
  • Iterative Data Collection: To train these models efficiently, the authors use an iterative loop. The planner attempts to solve training tasks using its current (possibly flawed) SEMs. The resulting plans are executed in simulation to collect ground-truth data, which is then used to update the SEMs.
  • Flexible Search: Because the planner searches directly in the space of continuous states and skill-parameter tuples, adding a new skill is as simple as adding a new branching option during the A* search. The planner evaluates the goal condition at the end of each predicted skill execution.

Datasets

The authors evaluate their method on a custom block manipulation domain built in the Nvidia Isaac Gym simulator, featuring a Franka Emika Panda 7-DoF arm. The environment includes blocks, a table, a tray, and a bin. They define four tasks (e.g., "Move All Blocks to Bin", "Move Red Blocks to Far Bin") split into training and testing sets. They also deploy the system in a physical real-world replica of this environment to test sim-to-real transfer.

Key Insights and Ideas

  • Macro-models beat micro-simulations: Predicting the outcome of an entire skill execution in one go (via SEMs) drastically reduces the planning horizon compared to step-by-step simulation. This bypasses the massive branching factor of low-level control and makes search-based planning computationally feasible.
  • Planner-guided exploration is highly sample-efficient: Training the SEMs on data generated by the planner's own attempts to solve tasks leads to much higher success rates with fewer samples than training on random skill executions. The planner naturally acts as a funnel, directing the learning toward the most useful regions of the state space.
  • Zero-shot task generalization: Because the system doesn't rely on learned "plan skeletons" (fixed sequences of skills), it can immediately apply newly learned skills to entirely unseen tasks just by changing the goal condition in the A* search.

Key Results

  • Massive Speedup: Planning with learned SEMs takes roughly 1 second, whereas using the ground-truth simulator for planning takes anywhere from 10 to 30 minutes.
  • Lifelong Skill Integration: As new skills (like Tray Slide, Tray Sweep, or Bin Tilt) were incrementally added to the robot's repertoire, the planner successfully found new, lower-cost strategies for both training and testing tasks. For example, adding the Tray Slide skill boosted the success rate on the difficult "Far Bin" tasks from 0% to 100%.
  • Real-World Transfer: The SEMs trained entirely in simulation successfully transferred to the physical robot without any fine-tuning, achieving high success rates (80-100%) on the real-world test tasks.

Limitations and Future Work

  • Parameter Sampling: The current planner uses random sampling to generate valid parameters for skills during the search. For skills with massive parameter spaces, this could become a bottleneck. The authors suggest using learned parameter samplers and partial node expansions in the future.
  • Uncertainty Estimation: The SEMs currently output deterministic predictions. Future work will explore estimating model uncertainty to steer the planner away from unknown regions, or to actively target fine-tuning where the model is least confident.
  • Preconditions: The main framework assumes skill preconditions (when a skill is valid to use) are known/given, though the authors do demonstrate a successful proof-of-concept for learning these preconditions in the appendix.

Summary updated

Abstract

Robots deployed in many real-world settings need to be able to acquire new skills and solve new tasks over time. Prior works on planning with skills often make assumptions on the structure of skills and tasks, such as subgoal skills, shared skill implementations, or task-specific plan skeletons, which limit adaptation to new skills and tasks. By contrast, we propose doing task planning by jointly searching in the space of parameterized skills using high-level skill effect models learned in simulation. We use an iterative training procedure to efficiently generate relevant data to train such models. Our approach allows flexible skill parameterizations and task specifications to facilitate lifelong learning in general-purpose domains. Experiments demonstrate the ability of our planner to integrate new skills in a lifelong manner, finding new task strategies with lower costs in both train and test tasks. We additionally show that our method can transfer to the real world without further fine-tuning.
BibTeX
@inproceedings{liang2022search,
  author = {Jacky Liang and Mohit Sharma and Alex LaGrassa and Shivam Vats and Saumya Saxena and Oliver Kroemer},
  title = {{Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=iRzmLwPyPAt},
}

Loading related content...

Figure