Generalizable Task Planning through Representation Pretraining

Chen Wang, Danfei Xu, Li Fei-Fei

Abstract

Robots operating in domestic environments must generalize complex, multi-step tasks to a wide variety of unseen objects and layouts. Current methods often struggle with this "generalization gap," either requiring extensive manual modeling or prohibitive amounts of interaction data. We introduce GenTP, a framework that leverages representation pretraining on large-scale synthetic datasets to bridge this gap. By learning hybrid object-centric representations that combine symbolic grounding with geometric reconstruction, GenTP allows robots to plan tasks using a spatially-grounded transition model. Our approach demonstrates superior generalization to novel objects and significantly improved search efficiency compared to state-of-the-art baselines.

Introduction

The challenge of robotic generalization is exemplified by the "blue spatula" problem: a robot trained to flip a pancake with one specific tool often fails when presented with a slightly different utensil or a new counter layout. Existing solutions typically fall into two categories: Task and Motion Planning (TAMP), which relies on rigid hand-coded logic, and End-to-End Learning, which requires massive interaction datasets. GenTP addresses this by utilizing "passive" scene-understanding data to build flexible, object-centric representations suitable for planning.

The GenTP Framework

The GenTP framework consists of three primary stages: representation pretraining, learning to plan via exploration, and generalization to new tasks.

![Overview of the GenTP framework:

  1. Representation pre-training, 2. Learning state transitions through exploration, and 3. Generalizing to new tasks with unseen objects.](/figures/art_117575f482514278aa75738a60193c70) Generalizable Task Planning through Representation Pretraining, page 1

1. Representation Pretraining

The core of GenTP is a hybrid representation learned from static synthetic scenes. An encoder is trained to map object instances to a latent space that supports both symbolic grounding (e.g., state classification like "is the towel soaked?") and geometric reconstruction (e.g., segmentation masks). This ensures the model understands both the semantic state and the physical shape/location of objects.

Detailed architecture of the representation learning stage, showing the encoder producing a latent representation used for both symbolic decoding and segmentation reconstruction.

2. Spatially-Grounded Planning

Once representations are fixed, a dynamics model ($f_T$) is trained to predict environment changes based on skills like pick, place, or pour. These skills are "grounded" in pixel space; instead of abstract commands, the model predicts the outcome of an action performed at specific $(x, y)$ coordinates.

3. Search-Based Execution

During task execution, the robot employs Breadth-First Search (BFS) to identify the optimal sequence of actions. To maintain efficiency, the learned symbolic decoder provides a confidence signal used to prune the search tree, focusing only on actions likely to result in meaningful state changes. This reduces the search space from millions of pixels to a handful of high-probability locations.

Experimental Evaluation

We evaluated GenTP using the BEHAVIOR benchmark and a custom synthetic scene dataset featuring diverse household objects (20 food types, 13 household objects). A small experience dataset of 500 trajectories was used to train the transition model, which was then tested on an unseen evaluation set of objects never encountered during interaction training.

Key Results

  • Superior Generalization: GenTP outperformed the state-of-the-art Deep Affordance Foresight (DAF) by over 10% on cooking tasks involving unseen objects. Baselines without pretraining dropped to near 0% success in these scenarios.
  • Sample Efficiency: The framework achieved higher success rates using only 60% of the training data required by baselines.
  • Search Speed: The pruning strategy resulted in an 80x speedup in planning time for 5-step tasks (79 seconds vs. >1000 seconds for standard pixel-wise search).

Conclusion and Future Work

GenTP demonstrates that pretraining object-centric representations on static scenes provides a powerful foundation for dynamic task planning. While currently dependent on high-quality segmentation masks and simplified skill parameterization (2D/3D points), the framework offers a scalable path toward generalizable domestic robots. Future work will focus on handling noisy real-world inputs and more complex manipulation skills like opening drawers or using orientation-sensitive tools.

Summary updated

Abstract

The ability to plan for multi-step manipulation tasks in unseen situations is crucial for future home robots. But collecting sufficient experience data for end-to-end learning is often infeasible in the real world, as deploying robots in many environments can be prohibitively expensive. On the other hand, large-scale scene understanding datasets contain diverse and rich semantic and geometric information. But how to leverage such information for manipulation remains an open problem. In this paper, we propose a learning-to-plan method that can generalize to new object instances by leveraging object-level representations extracted from a synthetic scene understanding dataset. We evaluate our method with a suite of challenging multi-step manipulation tasks inspired by the BEHAVIOR household activities and show that our model achieves measurably better success rate than state-of-the-art end-to-end approaches. Additional information can be found at: https://sites.google.com/view/gentp
BibTeX
@inproceedings{wang2022generalizable,
  author = {Chen Wang and Danfei Xu and Li Fei-Fei},
  title = {{Generalizable Task Planning through Representation Pretraining}},
  booktitle = {Long Horizon Planning @ CoRL},
  year = {2022},
  url = {https://openreview.net/forum?id=YiBS-duHVJ},
}
Figure