Make a Donut: Hierarchical EMD-Space Planning for Zero-Shot Deformable Manipulation With Tools

Yang You, Bokui Shen, Congyue Deng, Haoran Geng, Songlin Wei, He Wang, Leonidas J. GuibasIEEE RA-L 2025 · OpenReview

TL;DR

This paper introduces a novel, demonstration-free approach to robotic manipulation of deformable objects like dough. It leverages Large Language Models (LLMs) to break down complex tasks into stages and generate intermediate goals, which are then executed by a differentiable physics-based planner, enabling zero-shot generalization to new tasks.

What Problem Does This Paper Address?

Manipulating deformable objects, such as dough, is notoriously challenging for robots due to their unpredictable behavior. Existing methods often rely on learning from numerous demonstrations, which are difficult to collect, especially for long-horizon tasks. This reliance also limits the robot's ability to generalize to tasks it hasn't been explicitly trained on. The paper aims to overcome these limitations by developing a planning system that can tackle complex, multi-stage deformable manipulation tasks without requiring any prior demonstrations or task-specific training.

Methods

The proposed method employs a hierarchical planning strategy:

  1. High-Level LLM Planning: A Large Language Model (LLM), specifically GPT-4, is used to decompose a given natural language task (e.g., "Make a donut") into a sequence of stages. For each stage, the LLM identifies the appropriate tool and generates Python code to create a target point cloud representing the intermediate subgoal. This code leverages libraries like NumPy for point cloud generation. The LLM is guided by prompts that include available tools and general task semantics, but crucially, no task-specific demonstrations.
  2. Low-Level EMD-Space Planning: For each stage, a closed-loop model predictive control (MPC) strategy is employed. This planner takes the current state of the dough (represented as a point cloud) and the LLM-generated subgoal point cloud. It then uses a differentiable physics simulator (DiffPhysics-P2P) within the Earth Mover Distance (EMD) space to iteratively determine the next reachable state. The core of this is an EMD gradient descent that moves the current point cloud towards the target.
  3. Tool Reset and Initial Position Selection: To escape local minima and improve robustness, the system incorporates a tool reset mechanism if no progress is made. Additionally, it intelligently selects an optimal initial position for the tool at the start of each stage.

Datasets

The paper does not introduce new datasets for training. Instead, it evaluates its method on existing benchmarks and tasks:

  • Simulation: The method is evaluated on simulated dough manipulation tasks, including:
  • Multiple-tool tasks: Donut, Baguette, TwoPancakes.
  • Single-tool tasks: Spread, Cut, Arrange. These tasks are set up in environments consistent with prior work like PASTA [4] and utilize differentiable physics simulators such as DiffTaichi [13] and PlasticineLab [5].

  • Real-World: Experiments are conducted on a real robot (UFACTORY xArm 6) using clay.

Key Insights and Ideas

  • LLMs as Task Decomposers for Robotics: The paper demonstrates that LLMs can effectively break down complex, long-horizon robotic manipulation tasks into manageable, single-tool stages, a capability not fully exploited by prior LLM-for-robotics work which often focused on high-level instructions.
  • Zero-Shot Generalization via LLM-Generated Subgoals: By using LLMs to define intermediate goals (as point clouds), the system can generalize to novel tasks without any prior demonstrations. The LLM provides the "what," and the differentiable planner provides the "how."
  • EMD Space for Deformable Object Control: Utilizing the Earth Mover Distance (EMD) space for planning, particularly with a point-to-point correspondence, offers a robust way to measure and minimize the difference between the current and target states of deformable objects.
  • Bridging High-Level Intent and Low-Level Control: The hierarchical approach effectively bridges the gap between abstract task goals (from LLMs) and concrete, low-level physical actions, a critical challenge in complex manipulation.
  • Demonstration-Free Learning: The complete elimination of demonstration data for training is a significant step towards more adaptable and scalable robotic manipulation systems.

Key Results

  • Superior Performance: The proposed method significantly outperforms existing baselines (Diff. Physics, PASTA, BC, SAC-N) on both single-tool and multiple-tool deformable manipulation tasks, achieving up to 100% success rates in some cases.
  • Zero-Shot Success: The method demonstrates robust generalization to novel, complex tasks without any prior demonstrations or task-specific training. For instance, it successfully creates donuts, baguettes, and pancakes.
  • Real-World Validation: The approach was successfully translated to a real-world robot, demonstrating its practical applicability.
  • Efficiency: The planning process is relatively efficient, with a planning time per action of approximately 0.34s, enabling real-time control at around 3Hz.

Limitations and Future Work

  • Shape Generation Limitations: The current method is limited to generating simple shapes that can be described and constructed using Python code generated by the LLM. For more complex shapes that are difficult to describe programmatically (e.g., a bowl), the LLM needs to output text descriptions, which are then interpreted by external text-to-3D models (like Point-E).
  • LLM Output Quality: While LLMs are proficient at task decomposition, their generated subgoals might not always be perfectly reachable, requiring the closed-loop planner to compensate.
  • Future Work: The authors suggest extending the method to handle more complex shapes and potentially integrating more sophisticated text-to-3D generation capabilities directly into the LLM's output. Further exploration of LLM's role in generating more nuanced and precise low-level actions could also be a direction.

Summary updated

Abstract

Deformable object manipulation stands as one of the most captivating yet formidable challenges in robotics. While previous techniques have predominantly relied on learning latent dynamics through demonstrations, typically represented as either particles or images, there exists a pertinent limitation: acquiring suitable demonstrations, especially for long-horizon tasks, can be elusive. Moreover, basing learning entirely on demonstrations can hamper the model's ability to generalize beyond the demonstrated tasks. In this work, we introduce a demonstration-free hierarchical planning approach capable of tackling intricate long-horizon deformable manipulation tasks without necessitating any training. We employ large language models (LLMs) to articulate a high-level, stage-by-stage plan corresponding to a specified task. For every individual stage, the LLM provides both the tool's name and the Python code to craft intermediate subgoal point clouds. With the tool and subgoal for a particular stage at our disposal, we present a granular closed-loop model predictive control strategy. This leverages Differentiable Physics with Point-to-Point correspondence (DiffPhysics-P2P) loss in the Earth Mover Distance (EMD) space, applied iteratively. Experimental findings affirm that our technique surpasses multiple benchmarks in dough manipulation, spanning both short and long horizons. Remarkably, our model demonstrates robust generalization capabilities to novel and previously unencountered complex tasks without any preliminary demonstrations. We further substantiate our approach with experimental trials on real-world robotic platforms.
BibTeX
@article{you2025make,
  author = {Yang You and Bokui Shen and Congyue Deng and Haoran Geng and Songlin Wei and He Wang and Leonidas J. Guibas},
  title = {{Make a Donut: Hierarchical EMD-Space Planning for Zero-Shot Deformable Manipulation With Tools}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2025},
  url = {https://openreview.net/forum?id=4pqwp3kJW5},
}
Figure