DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability

Xiaolin Fang, Caelan Reed Garrett, Clemens Eppner, Tomás Lozano-Pérez, Leslie Pack Kaelbling, Dieter Fox

TL;DR

DiMSam (Diffusion Models as Samplers) integrates generative diffusion models into a Task and Motion Planning (TAMP) framework to solve long-horizon manipulation tasks under partial observability. By treating diffusion models as learned samplers for complex constraints, the system can reason about multi-step plans for unseen articulated objects without requiring explicit kinematic models.

What Problem Does This Paper Address?

Autonomous robot manipulation in the real world is notoriously difficult because it requires balancing two conflicting needs: long-horizon reasoning and low-level physical interaction.

Traditional Task and Motion Planning (TAMP) is excellent at reasoning about sequences of actions (e.g., "open the door, then put the bowl inside"), but it usually requires perfect knowledge of object geometry and kinematics to engineer the "samplers" that find valid robot trajectories. Conversely, End-to-End Learning (like RL) can handle raw sensor data and complex dynamics but often fails as the task horizon grows or the action space becomes high-dimensional.

DiMSam addresses the gap where a robot must manipulate an unfamiliar articulated object (like a new microwave model) using only partial point cloud observations, requiring both high-level logic and precise, contact-rich motion.

Methods

The core innovation is using Diffusion Models as the bridge between high-level planning and low-level execution.

  1. Latent Encoding: The system uses a PointNet++ encoder to map segmented partial point clouds ($\omega$) into a latent embedding ($z$). This allows the robot to represent the state of an object without knowing its exact 3D model.
  2. Diffusion Trajectory Sampler (DiffPush): Instead of a standard policy, the authors train a diffusion model to represent a distribution of trajectories. These trajectories consist of sequences of robot configurations ($q$) and latent object states ($z$).
  3. Classifier-Guided Sampling: To satisfy specific constraints (e.g., "the door must be closed"), they train small MLP classifiers. During the diffusion denoising process, the gradient of these classifiers guides the sampler toward valid, goal-satisfying trajectories.
  4. TAMP Integration: The learned diffusion models are plugged into PDDLStream, a TAMP solver. The solver searches for a high-level plan and calls the diffusion samplers to provide the continuous parameter values (like waypoints) that make the plan physically feasible.
Fig. 1: DIMSAM composes diffusion models using a TAMP system towards solving multi-step manipulation problems. The planner searches for the task plan, while learned diffusion samplers find constraint-satisfying continuous values. The bottom shows a sampling procedure for finding a microwave door closing trajectory and a collision-free object stowing trajectory. A diffusion model samples a trajectory of latent microwave states z1, ..., zT and robot configurations q1, ..., qT that reaches a door-closed state zT .

Datasets

The models were trained and evaluated primarily in the IsaacGym physics simulator:

  • Assets: 10 different microwave models from the PartNet dataset.
  • Training Data: 101 valid trajectories (later clipped into ~4,700 segments) generated by a script-based policy with access to ground-truth physics.
  • Real-World Testing: The system was tested on a physical Franka Emika robot manipulating a real microwave, using zero-shot transfer (no fine-tuning on real-world data).

Key Insights and Ideas

  • Diffusion > Baselines: Diffusion models proved superior to Energy-Based Models (EBMs) and standard regression. They are "diverse" (can generate many different valid paths) and "compositional" (can be combined with new classifiers after training).
  • Soundness through TAMP: While generative models can sometimes produce "hallucinations," embedding them within a TAMP framework ensures that the final plan is logically sound and satisfies all specified constraints, such as collision avoidance.
  • Latent Space Generalization: By planning in a latent space derived from point clouds, the system generalizes to objects it has never seen before, effectively "imagining" how the object's state will change based on its interaction.
  • Zero-Shot Sim-to-Real: Because the diffusion model outputs end-effector waypoints rather than raw motor torques, the system is highly portable. The authors successfully transferred the simulation-trained models to a real robot by simply swapping the Inverse Kinematics (IK) solver.
Fig. 3: Samples from the DiffPush model. (a) No con- dition except for known z1. (b) Checking collision with the purple obstacle using classifier PairwiseCollision. Rejected samples are colored red. (c) (d) Classifier-guided conditional sampling with DoorOpen and Doorclosed.

Key Results

  • Trajectory Accuracy: In simulation, the diffusion sampler achieved a mean error of 0.094 radians relative to the target door angle, significantly outperforming the EBM (0.191) and regression (0.127) baselines.
  • High-Level Success: The integrated TAMP system achieved a 100% execution success rate on complex "Stow-Close" tasks when using conditional samplers.
  • Reasoning Capability: In the "Stow-Close-B" task, the planner correctly identified that it needed to move an obstructing object before it could open the microwave door to stow a large bowl.
  • Real-World Robustness: The system successfully performed door opening and closing on a real microwave, even when the depth camera provided noisy, incomplete data due to the microwave's reflective surfaces.

Limitations and Future Work

  • Kinematic Feasibility: Occasionally, the diffusion model samples trajectories that are too close to the joint axis of the door, making them physically difficult for the robot to execute. The authors suggest incorporating "effort estimates" or uncertainty into future samplers.
  • Combinatorial Explosion: As the number of required actions increases, the TAMP search space grows exponentially. Improving the efficiency of the joint search and sampling process remains a priority.
  • Object Diversity: While the system excels at articulated objects like microwaves, extending this to a wider variety of household objects and more complex contact dynamics is the next step.

Summary updated

Abstract

Generative models such as diffusion models, excel at capturing high-dimensional distributions with diverse input modalities, e.g. robot trajectories, but are less effective at multi-step constraint reasoning. Task and Motion Planning (TAMP) approaches are suited for planning multi-step autonomous robot manipulation. However, it can be difficult to apply them to domains where the environment and its dynamics are not fully known. We propose to overcome these limitations by composing diffusion models using a TAMP system. We use the learned components for constraints and samplers that are difficult to engineer in the planning model, and use a TAMP solver to search for the task plan with constraint-satisfying action parameter values. To tractably make predictions for unseen objects in the environment, we define the learned samplers and TAMP operators on learned latent embedding of changing object states. We evaluate our approach in a simulated articulated object manipulation domain and show how the combination of classical TAMP, generative modeling, and latent embedding enables multi-step constraint-based reasoning. We also apply the learned sampler in the real world. Website: https://sites.google.com/view/dimsam-tamp
BibTeX
@inproceedings{fang2024dimsam,
  author = {Xiaolin Fang and Caelan Reed Garrett and Clemens Eppner and Tomás Lozano-Pérez and Leslie Pack Kaelbling and Dieter Fox},
  title = {{DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802746},
}
Figure