Visual Robot Task Planning

Chris Paxton, Yotam Barnoy, Kapil D. Katyal, Raman Arora, Gregory D. HagerICRA 2019 · DOI

TL;DR

This paper introduces a novel neural network architecture and planning algorithm for robots that learns to predict the consequences of high-level actions in visual environments. By learning an interpretable latent representation of the world, the system can simulate and evaluate potential future states, enabling it to plan complex tasks from visual input alone, even in novel situations.

What Problem Does This Paper Address?

Robots currently struggle with general-purpose task planning, especially in complex, visually rich environments. Traditional methods often require explicit, hand-engineered models of the world, object dynamics, and action effects, which is time-consuming and brittle. Humans, on the other hand, excel at "prospection" – predicting the outcomes of their actions – which is crucial for intelligent planning. This paper aims to bridge this gap by enabling robots to learn this prospection capability directly from visual observations and demonstrations, allowing them to plan complex activities without explicit task knowledge.

Methods

The core of the proposed approach is a neural network architecture that learns a latent representation of the world. This architecture consists of:

  • Encoder ($f_{enc}$): Maps visual observations (images) into a low-dimensional, interpretable hidden state ($h$).
  • Decoder ($f_{dec}$): Maps from the hidden state back to the visual observation space.
  • Transform Function ($T(h_o, h, a)$): This is the key novel component. It takes the previous hidden state ($h_o$), the current hidden state ($h$), and a high-level action ($a$) to predict the next hidden state ($h'$). This function is designed to capture the dynamics of the world and the effects of actions in the latent space.

The planning algorithm uses this learned model within a Monte Carlo Tree Search (MCTS) framework. The MCTS explores possible sequences of high-level actions, using the learned transform function to simulate future states and value functions ($V(h)$ and $Q(h_o, h, a, a')$) to evaluate the likelihood of success for each action sequence. This allows the robot to "prospective" and choose the most promising plan.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Simulated Husky Robot Navigation: A custom dataset generated in a Gazebo simulation for a navigation task.
  • Simulated UR5 Block Stacking: A custom dataset generated in PyBullet for a block-stacking task, including an obstacle.
  • JIGSAWS Dataset: A subset of the JIGSAWS dataset for surgical tasks (suturing, knot tying, needle passing), using stereo video frames.

Key Insights and Ideas

  • Prospection as a Core for Robot Planning: The paper frames robot planning through the lens of human "prospection," suggesting that predicting future consequences is fundamental to intelligence.
  • Learning a Latent World Model: Instead of relying on explicit physics or task models, the system learns a compressed, interpretable latent representation of the world and its dynamics.
  • High-Level Action Simulation: The learned transform function allows for the simulation of sequences of high-level actions in this latent space, enabling planning without needing to execute low-level motor commands directly.
  • Interpretable Planning: The latent space and the ability to visualize predicted futures offer a degree of interpretability, allowing the system to explain why a plan might fail.
  • Generalization to Novel Environments: The learned models are trained on imperfect demonstrations and are shown to generalize to new, unseen environments.

Key Results

  • The proposed model successfully learned to predict future states and plan tasks in simulated navigation, block stacking, and real-world surgical scenarios.
  • In the block-stacking task, the system was able to find successful plans in new environments, and in cases of failure, it could provide explanations (e.g., proximity to an obstacle).
  • The value function correctly identified successful or failed trials with high accuracy (83.9%) and classified possible next actions with 96.0% accuracy.
  • Experiments showed that including a classifier loss term improved the quality and crispness of predictions, particularly for capturing fine details.
  • The "No Skips" architecture (without skip connections in the transform block) performed comparably, suggesting that while skip connections help with background detail, they are not essential for core task aspects.

Limitations and Future Work

  • Data Requirements: While the approach uses imperfect demonstrations, it still requires a significant amount of training data, as seen in the 100,000 demonstrations used in prior work for block stacking (though this paper used fewer).
  • Pixel-wise Error: While the learned latent space is expressive, the decoder might lose some fine details, and the transform function can accumulate pixel-wise errors over many steps.
  • Surgical Task Data: The surgical task experiments were limited by a relatively small amount of available data, which impacted performance compared to robotic manipulation tasks.
  • Future Work: The authors plan to apply the method to real robotic examples and expand experiments on surgical data. They also aim to further improve the expressive power of the learned representation and explore more complex planning scenarios.

Summary updated

BibTeX
@inproceedings{paxton2019visual,
  author = {Chris Paxton and Yotam Barnoy and Kapil D. Katyal and Raman Arora and Gregory D. Hager},
  title = {{Visual Robot Task Planning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  doi = {10.1109/icra.2019.8793736},
}
Figure