Introduction
Robotics research has traditionally been bifurcated into two distinct paradigms: deep learning and symbolic planning. Deep learning excels at processing high-dimensional sensory data, such as raw pixels, but often struggles with long-horizon tasks and generalization to novel sequences. Conversely, symbolic planning offers mathematical precision for complex, multi-step problem solving but relies on abstract symbols that are difficult to "ground" in the noisy, visual reality of the physical world.
Researchers from NVIDIA and Waseda University have introduced Deep Planning Domain Learning (DPDL), a hierarchical framework designed to bridge this gap. By combining neural networks for visual perception with symbolic logic for high-level reasoning, DPDL enables robots to perform complex tasks in photorealistic environments with a high degree of transferability and reactivity.
The DPDL Framework
DPDL utilizes a hierarchical architecture powered by Variational Temporal Convolutional Networks (V-TCNs). The system is divided into two primary layers that separate high-level reasoning from low-level motor control.
High-Level Network: The "Thinker"
The high-level network processes sequences of RGB images and joint angles to predict the current logical world state. Rather than outputting direct actions, it identifies a set of boolean predicates (e.g., ArmIsNearHandle, ObjIsOnCounter). This grounded state is then passed to a symbolic planner (such as STRIPS or PDDL), which determines the optimal sequence of operators to reach a goal.
Low-Level Network: The "Doer"
The low-level network acts as a conditional policy. Once the symbolic planner selects an operator (e.g., Grasp Obj), this network translates the current visual observations and the specific command into a target joint configuration for the robot arm.
Symbol Grounding and Auxiliary Tasks
A central challenge in this architecture is the symbol grounding problem: accurately mapping raw pixels to logical states. To address this, the researchers employed "auxiliary losses" during training. The model is forced to predict depth maps and semantic segmentation masks alongside logical predicates. This ensures the network learns the underlying 3D structure and object identities of the scene, which are critical for robust grounding, especially under occlusion.
Experimental Results and Reactivity
The system was evaluated using a 7-DoF Franka Emika Panda robot in a photorealistic kitchen simulation. The training data consisted of simple tasks like "put away sugar," yet the system demonstrated remarkable zero-shot recombination. Because it learns individual operators rather than fixed sequences, the symbolic planner can generate paths for novel, multi-step tasks that were never seen during training.
Key performance metrics include:
- Generalization: DPDL achieved an 85% success rate on the "Put Away Sugar" task, while a standard end-to-end V-TCN baseline failed completely (0% success).
- Robustness: In a "teleportation test" where objects were moved mid-task, DPDL’s reactive loop allowed it to recover and succeed 85% of the time, compared to only 30% for non-hierarchical models.
- Occlusion Handling: The predicate-based approach significantly outperformed state-of-the-art pose estimators like PoseCNN when objects were partially hidden inside drawers or cabinets.
Conclusion
DPDL demonstrates that a hierarchical approach—separating visual grounding from logical planning—is key to achieving transferable task execution in robotics. By grounding symbols through auxiliary visual tasks, the system gains the robustness of deep learning and the reasoning power of symbolic logic. Future work aims to replace formal logic goals with natural language commands and to validate the framework on physical hardware in real-world settings.
Summary updated