Transferable Task Execution from Pixels through Deep Planning Domain Learning

Kei Kase, Chris Paxton, Hammad Mazhar, Tetsuya Ogata, Dieter Fox

Introduction

Robotics research has traditionally been bifurcated into two distinct paradigms: deep learning and symbolic planning. Deep learning excels at processing high-dimensional sensory data, such as raw pixels, but often struggles with long-horizon tasks and generalization to novel sequences. Conversely, symbolic planning offers mathematical precision for complex, multi-step problem solving but relies on abstract symbols that are difficult to "ground" in the noisy, visual reality of the physical world.

Researchers from NVIDIA and Waseda University have introduced Deep Planning Domain Learning (DPDL), a hierarchical framework designed to bridge this gap. By combining neural networks for visual perception with symbolic logic for high-level reasoning, DPDL enables robots to perform complex tasks in photorealistic environments with a high degree of transferability and reactivity.

The DPDL Framework

DPDL utilizes a hierarchical architecture powered by Variational Temporal Convolutional Networks (V-TCNs). The system is divided into two primary layers that separate high-level reasoning from low-level motor control.

The DPDL hierarchical architecture, showing the interaction between the High-Level Network (predicates) and the Low-Level Network (target joints).

High-Level Network: The "Thinker"

The high-level network processes sequences of RGB images and joint angles to predict the current logical world state. Rather than outputting direct actions, it identifies a set of boolean predicates (e.g., ArmIsNearHandle, ObjIsOnCounter). This grounded state is then passed to a symbolic planner (such as STRIPS or PDDL), which determines the optimal sequence of operators to reach a goal.

Low-Level Network: The "Doer"

The low-level network acts as a conditional policy. Once the symbolic planner selects an operator (e.g., Grasp Obj), this network translates the current visual observations and the specific command into a target joint configuration for the robot arm.

Symbol Grounding and Auxiliary Tasks

A central challenge in this architecture is the symbol grounding problem: accurately mapping raw pixels to logical states. To address this, the researchers employed "auxiliary losses" during training. The model is forced to predict depth maps and semantic segmentation masks alongside logical predicates. This ensures the network learns the underlying 3D structure and object identities of the scene, which are critical for robust grounding, especially under occlusion.

An annotated view of the photorealistic kitchen environment, illustrating the various objects and states (predicates) the robot must ground from pixels.

Experimental Results and Reactivity

The system was evaluated using a 7-DoF Franka Emika Panda robot in a photorealistic kitchen simulation. The training data consisted of simple tasks like "put away sugar," yet the system demonstrated remarkable zero-shot recombination. Because it learns individual operators rather than fixed sequences, the symbolic planner can generate paths for novel, multi-step tasks that were never seen during training.

Key performance metrics include:

  • Generalization: DPDL achieved an 85% success rate on the "Put Away Sugar" task, while a standard end-to-end V-TCN baseline failed completely (0% success).
  • Robustness: In a "teleportation test" where objects were moved mid-task, DPDL’s reactive loop allowed it to recover and succeed 85% of the time, compared to only 30% for non-hierarchical models.
  • Occlusion Handling: The predicate-based approach significantly outperformed state-of-the-art pose estimators like PoseCNN when objects were partially hidden inside drawers or cabinets.

Conclusion

DPDL demonstrates that a hierarchical approach—separating visual grounding from logical planning—is key to achieving transferable task execution in robotics. By grounding symbols through auxiliary visual tasks, the system gains the robustness of deep learning and the reasoning power of symbolic logic. Future work aims to replace formal logic goals with natural language commands and to validate the framework on physical hardware in real-world settings.

Summary updated

Abstract

While robots can learn models to solve many manipulation tasks from raw visual input, they cannot usually use these models to solve new problems. On the other hand, symbolic planning methods such as STRIPS have long been able to solve new problems given only a domain definition and a symbolic goal, but these approaches often struggle on the real world robotic tasks due to the challenges of grounding these symbols from sensor data in a partially-observable world. We propose Deep Planning Domain Learning (DPDL), an approach that combines the strengths of both methods to learn a hierarchical model. DPDL learns a high-level model which predicts values for a large set of logical predicates consisting of the current symbolic world state, and separately learns a low-level policy which translates symbolic operators into executable actions on the robot. This allows us to perform complex, multistep tasks even when the robot has not been explicitly trained on them. We show our method on manipulation tasks in a photorealistic kitchen scenario.
BibTeX
@inproceedings{kase2020transferable,
  author = {Kei Kase and Chris Paxton and Hammad Mazhar and Tetsuya Ogata and Dieter Fox},
  title = {{Transferable Task Execution from Pixels through Deep Planning Domain Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=qkdE3Nflv5},
}
Figure