DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability

Xiaolin Fang, Caelan Garrett, Clemens Eppner, Tomás Lozano-Pérez, Leslie Kaelbling, Dieter Fox

Introduction

Robotic manipulation in unstructured environments requires a delicate balance between high-level logical reasoning and precise continuous control. Traditional Task and Motion Planning (TAMP) frameworks excel at long-horizon logic but struggle with "samplers"—functions that propose valid continuous trajectories for complex interactions or novel objects. Conversely, end-to-end learning methods often lack the compositional reasoning needed for multi-step tasks.

Overview of the DiMSam framework solving a multi-step manipulation task involving a bowl and a microwave.

DiMSam (Diffusion Models as Samplers) addresses these challenges by integrating diffusion models into a TAMP system. By leveraging diffusion to sample valid robot trajectories and latent object states, DiMSam enables robots to solve complex manipulation tasks involving unseen articulated objects under partial observability.

Methodology: Diffusion as a Smart Sampler

The core of DiMSam is the integration of a Diffusion Trajectory Sampler within the PDDLStream TAMP framework. This hybrid approach treats the neural network not as a monolithic policy, but as a "smart suggestion engine" for a classical planner.

The DiMSam system architecture, showing the flow from raw point cloud observations to task planning and diffusion-based trajectory sampling.

The system operates through four primary components:

  1. Latent Parameter Encoding: To generalize across diverse object geometries, a PointNet++ encoder transforms partial point clouds into a compact latent state ($z$). This allows the robot to plan based on the effect of its actions rather than requiring explicit kinematic models.
  2. Diffusion Trajectory Sampler: A diffusion model is trained to generate sequences of robot configurations ($q$) and latent states ($z$). Its bi-directional nature allows it to find paths between specific start and end states.
  3. Classifier-Guided Sampling: Separate neural network classifiers provide gradients to nudge the diffusion process toward trajectories that satisfy specific constraints, such as avoiding collisions or ensuring a door reaches a closed state.
  4. TAMP Integration: The high-level planner searches for logical action sequences and calls the Diffusion Sampler to provide the continuous execution parameters for each step.

Key Results and Insights

DiMSam was evaluated in both simulation (using PartNet assets in IsaacGym) and real-world environments, yielding several key insights:

  • High Precision and Success: The diffusion-based sampler reached target door angles with significantly lower error (0.094 radians) compared to standard regression or energy-based models. The system demonstrated a 94% success rate in closing doors and a 92% success rate in opening them using classifier guidance.
  • Complex Reasoning: In a "Stow-Close" task where an obstacle blocked the microwave, the TAMP + DiMSam system correctly inferred it had to move the obstacle first, then open the door, and finally stow the object.
  • Zero-Shot Sim-to-Real: The model successfully transferred to a physical Franka Emika robot. It handled noisy point clouds from a RealSense camera—even with reflective surfaces—without additional fine-tuning.

Conclusion

By using diffusion models as samplers, DiMSam combines the reliability of logical planning with the flexibility of deep learning. The modularity of classifier-guided sampling allows for the composition of constraints without retraining the core model. While challenges remain regarding the computational cost of long-horizon searches and occasional physical infeasibilities, DiMSam offers a promising path toward more generalizable and robust robotic manipulation.

Summary updated

Abstract

Task and Motion Planning (TAMP) approaches are effective at planning long-horizon autonomous robot manipulation. However, it can be difficult to apply them to domains where the environment and its dynamics are not fully known. We propose to overcome these limitations by leveraging deep generative modeling, specifically diffusion models, to learn constraints and samplers that capture these difficult-to-engineer aspects of the planning model. These learned samplers are composed and combined within a TAMP solver in order to find action parameter values jointly that satisfy the constraints along a plan. To tractably make predictions for unseen objects in the environment, we define these samplers on low-dimensional learned latent embeddings of changing object state. We evaluate our approach in an articulated object manipulation domain and show how the combination of classical TAMP, generative learning, and latent embeddings enables long-horizon constraint-based reasoning. We also apply the learned sampler in the real world. More details are available at https://sites.google.com/view/dimsam-tamp.
BibTeX
@inproceedings{fang2024dimsam,
  author = {Xiaolin Fang and Caelan Garrett and Clemens Eppner and Tomás Lozano-Pérez and Leslie Kaelbling and Dieter Fox},
  title = {{DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802746},
}
Figure