Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objects

Cheng Chi, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, Shuran SongInt. J. Robotics Res 2022 · DOI · OpenReview

TL;DR

This paper introduces the Iterative Residual Policy (IRP), a novel framework for teaching robots to manipulate deformable objects (like ropes and cloths) with high precision, even when dealing with complex dynamics and unknown object properties. IRP iteratively refines actions based on visual feedback and a learned "delta dynamics" model, enabling robust generalization to unseen objects and real-world conditions.

What Problem Does This Paper Address?

Robots struggle with manipulating deformable objects because their behavior is inherently complex and hard to model. Unlike rigid objects, their deformation, high-speed interactions, and subtle physical properties (like stiffness and friction) introduce non-linear dynamics that are difficult to predict. This makes tasks requiring precise control, such as whipping a rope to hit a specific point or arranging a cloth into a target pose, extremely challenging for current robotic systems. Existing methods often rely on quasi-static manipulation (slow and deliberate) or require extensive retraining for new objects, limiting their applicability to real-world scenarios where speed and adaptability are crucial.

Methods

The core of the proposed solution is the Iterative Residual Policy (IRP). Instead of trying to model the entire complex dynamics of the object and action, IRP focuses on learning "delta dynamics." This means the system learns to predict how a small change in action will affect the object's trajectory, given the current state.

Here's how it works:

  1. Initial Action Guess: The robot starts with an initial, best-guess action.
  2. Observe Trajectory: The robot executes the action and its resulting trajectory is observed (e.g., the path of a rope tip).
  3. Sample Delta Actions: A set of small "delta actions" (perturbations to the current action) are sampled.
  4. Predict Delta Dynamics: A learned neural network (specifically, a DeepLabV3+ architecture adapted for this task) predicts the resulting trajectories for each sampled delta action. This network takes the observed trajectory and the delta action as input.
  5. Select Best Delta Action: The delta action that is predicted to bring the object closest to the goal is greedily selected.
  6. Refine Action: The selected delta action is added to the current action, creating a refined action for the next iteration.
  7. Repeat: Steps 2-6 are repeated until the goal is reached or a maximum number of iterations is met.

This iterative refinement, combined with adaptive action sampling (where the sampling range is scaled by the current error), allows the system to quickly converge to precise goals and adapt to errors from the model, observations, and real-world noise.

Datasets

The paper primarily uses simulated data for training and real-world experiments for evaluation.

  • Simulation Data: Generated using MuJoCo for both rope whipping and cloth placement tasks.
  • Rope: Simulated with varying parameters like length and density, creating a dataset of 54 million trajectories.
  • Cloth: Simulated with varying size and density, resulting in 131 thousand examples.
  • Real-world Data: Used for evaluating the trained policy on physical robots. This includes testing with various real-world ropes and different robot hardware configurations.

Key Insights and Ideas

  • Delta Dynamics is Easier to Learn: Predicting the change in trajectory due to a small action change is significantly easier and more generalizable than modeling the entire forward dynamics. This is a key insight for handling complex, non-linear systems.
  • Iterative Refinement for Precision: For tasks requiring high precision, an iterative approach that gradually corrects actions is more effective than trying to find the perfect action in a single shot. This allows for robustness against model inaccuracies.
  • Generalization Beyond Simulation: The framework demonstrates remarkable generalization capabilities, performing well on unseen objects, different physical properties, and even different robot hardware, despite being trained solely in simulation. This highlights the power of the learned delta dynamics and adaptive sampling.
  • Bridging the Sim-to-Real Gap: The iterative, feedback-driven approach effectively bridges the sim-to-real gap, allowing a policy trained in simulation to perform well on real hardware without explicit sim-to-real domain adaptation techniques.

Key Results

  • High Precision: The IRP achieved pixel-level accuracy in simulation (1.8cm) and impressive accuracy in the real world (2.6cm for rope whipping).
  • Robust Generalization: The method generalized to unseen rope parameters (both interpolated and extrapolated), different materials, and even different robot end-effector lengths, outperforming baseline methods like system identification and heuristic control.
  • Efficiency: The iterative approach allowed for rapid convergence, often reaching goals in fewer steps than other iterative methods. For example, in real-world rope experiments, IRP achieved significantly lower distances to the goal in fewer iterations compared to iterLinear.
  • Cloth Manipulation Success: The framework was successfully extended to cloth manipulation, achieving an average error of 3.5 cm across various goals, outperforming DeltaReg and IterHeuristic.

Limitations and Future Work

  • Action Repeatability Assumption: The current method assumes that actions are repeatable. This might not hold in highly dynamic or non-deterministic scenarios.
  • Full Observability: The approach assumes full observability of the manipuland throughout the trajectory. This limits its direct application in cluttered environments where parts of the object might be occluded.
  • Future Work: The authors suggest exploring ways to handle non-repeatable action scenarios and partial observability, potentially by incorporating different visual feedback mechanisms or temporal cues.

Summary updated

Abstract

This paper tackles the task of goal-conditioned dynamic manipulation of deformable objects. This task is highly challenging due to its complex dynamics (introduced by object deformation and high-speed action) and strict task requirements (defined by a precise goal specification). To address these challenges, we present Iterative Residual Policy (IRP), a general learning framework applicable to repeatable tasks with complex dynamics. IRP learns an implicit policy via delta dynamics -- instead of modeling the entire dynamical system and inferring actions from that model, IRP learns delta dynamics that predict the effects of delta action on the previously-observed trajectory. When combined with adaptive action sampling, the system can quickly optimize its actions online to reach a specified goal. We demonstrate the effectiveness of IRP on two tasks: whipping a rope to hit a target point and swinging a cloth to reach a target pose. Despite being trained only in simulation on a fixed robot setup, IRP is able to efficiently generalize to noisy real-world dynamics, new objects with unseen physical properties, and even different robot hardware embodiments, demonstrating its excellent generalization capability relative to alternative approaches. Video is available at https://youtu.be/7h3SZ3La-oA
BibTeX
@article{chi2022iterative,
  author = {Cheng Chi and Benjamin Burchfiel and Eric Cousineau and Siyuan Feng and Shuran Song},
  title = {{Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objects}},
  journal = {Int. J. Robotics Res.},
  year = {2022},
  doi = {10.1177/02783649231201201},
}
Figure