Learning Representations for Deformable Object Manipulation

261 concepts200 papers1 proposal

When a robot manipulates a rigid object like a block or a tool, its state can be perfectly described by just a few numbers: a three-dimensional position and a three-dimensional orientation. This low-dimensional state space makes tracking, predicting, and planning straightforward.

Deformable objects—such as cloth, ropes, dough, and liquids—shatter this paradigm. Because soft materials can bend, stretch, crumple, and twist in an infinite number of ways, they possess effectively infinite degrees of freedom. If you attempt to represent a crumpled shirt by tracking every single molecule or vertex, the state space becomes too massive to navigate. Compounding this challenge is the issue of severe self-occlusion: as fabric folds or dough kneads, critical parts of the object disappear from visual sensors entirely.

As a result, traditional reinforcement learning algorithms often fail to converge on stable trajectories. As noted by Haoran Lu et al. in GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation, trying to solve this high-dimensional state space directly leads to unstable trajectories and physical failures, such as a robotic arm becoming entangled in the fabric.

To make deformable object manipulation tractable, modern research focuses on a shared core insight: we must compress the infinite-dimensional physical world into structured, control-oriented representations. Instead of tracking everything, robots must learn what to ignore.


Graph Dynamics: Capturing Topology and Physics

One of the most intuitive ways to simplify a deformable object is to represent it as a graph. Rather than treating a piece of cloth or dough as a collection of unorganized pixels or point clouds, a graph structure explicitly models the object as a set of keypoints (nodes) connected by physical relationships (edges). This captures the underlying topology of the material while ignoring superficial surface wrinkles.

[Pixel Input] ──► [Keypoint Extraction] ──► [Graph Neural Network] ──► [Action Prediction]
                                                   │
                                           (Learns Edges/Elasticity)

This approach allows a robot to infer hidden physical properties through interaction. For instance, Alberta Longhini et al. developed EDO-Net: Learning Elastic Properties of Deformable Objects from Graph Dynamics, a framework where a robot "feels" a fabric's elasticity via a simple tug. By mapping these interactions to a graph, the model learns a latent representation of the material's non-linear physical responses without requiring explicit labels, generalizing successfully to entirely unseen fabrics.

Graphs also excel at modeling complex tool-object interactions. In Learning Goal-oriented Bimanual Dough Rolling Using Dynamic Heterogeneous Graph Based on Human Demonstration, Junjia Liu et al. utilized dynamic heterogeneous graphs to represent both the deformable dough and the rigid rolling pin as interconnected nodes. Because the graph updates its edges dynamically as the dough deforms under pressure, the robot can accurately predict shape changes and transfer complex bimanual manipulation skills from human demonstrations to real-world hardware.


Differentiable Physics and Latent Planning

While graphs provide explicit structure, another powerful paradigm leverages differentiable physics simulators to optimize actions directly.

In traditional "black-box" simulations, a robot tries an action, observes the outcome, and uses trial-and-error to improve. Differentiable simulators, by contrast, allow the robot to compute exact analytical gradients through time. This means the robot can calculate precisely how a tiny change in its gripper's trajectory will alter the final shape of a soft object.

To plan over long time horizons, researchers combine these simulators with compressed latent spaces. In DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools, Xingyu Lin et al. demonstrated a system that learns short-term "skills" (like cutting or lifting dough) from raw visual inputs. By mapping high-dimensional images into a compact latent space, the framework uses the gradients from a differentiable simulator to chain these skills together, solving complex, multi-step tool use that standard reinforcement learning cannot handle.

[High-Dim Image] ──► [Latent Space Compression] ──► [Differentiable Sim Planning] ──► [Gradient-Based Trajectory]

When analytical gradients become too complex or noisy due to contact discontinuities, generative models can step in to structure the latent space. Jun Yamada et al. introduced D-Cubed: Latent Diffusion Trajectory Optimisation for Dexterous Deformable Manipulation, which trains a latent diffusion model on task-agnostic "play" data. By using a gradient-free optimization technique over this learned latent space, a multi-fingered robotic hand can generate complex trajectories to manipulate dough or cloth without ever needing an explicit analytical model of the material's geometry.


Bypassing State Estimation via Visual Correspondence

Reconstructing a full 3D mesh or graph is not always necessary—or even possible—when an object is highly crumpled. An alternative strategy bypasses explicit state estimation entirely by focusing on visual correspondence: mapping the spatial relationship between the current visual state and the goal visual state.

Instead of calculating where every part of a cloth is in 3D space, a robot can look at the pixel-level motion required to transform the current image into the target image. Thomas Weng et al. implemented this in FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy by utilizing optical flow. By calculating a dense flow field between the current cloth state and a goal image, the policy directly determines the optimal pick-and-place coordinates for a bimanual robot, significantly outperforming model-free alternatives.

[Current Image] ┐
                ├─► [Optical Flow / Correspondence] ─► [Pick & Place Action]
[Goal Image]    ┘

However, greedy visual matching can trap a robot in "local optima"—actions that look beneficial immediately but make future steps impossible (such as folding a cloth over a corner that needs to be reached later).

To inject foresight into visual policies, Ruihai Wu et al. proposed Learning Foresightful Dense Visual Affordance for Deformable Object Manipulation. Their framework pairs pixel-level action maps with long-term value estimation. By collecting data through a self-supervised "Fold to Unfold" strategy, the robot learns to anticipate the long-term geometric consequences of its actions, ensuring that early manipulation steps actively prepare the object for downstream success.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 201 papers
  2. Version 1 · 2026-10-04 · 427 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_deformable_object_manipulation_high_dime_5b62b9b8,
  title = {deformable object manipulation (high-dimensional state space)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deformable-object-manipulation-high-dimensional-state-space/rdr_5b62b9b8f687470986ecbbcea31eb98e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_deformable_object_manipulation_high_dime_5b62b9b8_v2,
  title = {deformable object manipulation (high-dimensional state space)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deformable-object-manipulation-high-dimensional-state-space/rdr_5b62b9b8f687470986ecbbcea31eb98e/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake