Can Real-to-Sim Approaches Capture Dynamic Fabric Behavior for Robotic Fabric Manipulation?

Yingdong Ru, Lipeng Zhuang, Zhuo He, Florent P. Audonnet, Gerardo Aragon-Camarasa

Abstract

This paper presents a rigorous evaluation of Real-to-Sim parameter estimation approaches for fabric manipulation in robotics. The study systematically assesses three state-of-the-art approaches, namely two differential pipelines and a data-driven approach. We also devise a novel physics-informed neural network approach for physics parameter estimation. These approaches are interfaced with two simulations across multiple Real-to-Sim scenarios (lifting, wind blowing, and stretching) for five different fabric types and evaluated on three unseen scenarios (folding, fling, and shaking). We found that the simulation engines and the choice of Real-to-Sim approaches significantly impact fabric manipulation performance in our evaluation scenarios. Moreover, PINN observes superior performance in quasi-static tasks but shows limitations in dynamic scenarios.

Introduction

Robotic manipulation of deformable objects, particularly fabric, remains one of the most significant challenges in the field. Unlike rigid objects, garments possess a high-dimensional configuration space and exhibit complex, non-linear behaviors under stress. To train effective robotic policies, researchers frequently turn to simulators. However, a persistent "Sim-to-Real gap" exists: the physical parameters within a simulator—such as bending stiffness, shearing, or friction—rarely align perfectly with the properties of real-world materials.

While "Real-to-Sim" (real2sim) methods have been developed to bridge this gap by aligning simulated behavior with real-world observations, they are often validated on the same scenarios used for training. This research investigates a critical question: Do these methods estimate physically meaningful parameters that generalize to new, unseen manipulation tasks, or are they merely overfitting to specific motions?

Methodology

The study systematically compares three categories of Real-to-Sim approaches across two distinct simulation engines: DiffSim (utilizing an anisotropic material model) and DiffTaichi (utilizing an isotropic model).

  1. Differentiable Pipelines (DIFFCLOUD and DiffCP): These methods employ gradient-based optimization to minimize the geometric distance between real-world and simulated point clouds.
  2. Data-Driven Approaches (PhysNet): This utilizes a Siamese neural network architecture to learn physical similarity clusters among fabrics using a triplet loss function.
  3. Physics-Informed Neural Networks (PINNs): The authors introduce a novel PINN framework for fabric. Unlike standard neural networks that learn purely from visual data, PINNs embed the governing Partial Differential Equations (PDEs) of elastodynamics directly into the loss function. This ensures the model respects fundamental physical laws—such as stress, strain, and acceleration—while learning from observed data.

Experimental Setup and Dataset

To facilitate this evaluation, the authors developed a comprehensive dataset (approximately 50 GB) featuring five distinct fabric types, ranging from highly deformable materials like Camel Ponte Roma to more rigid, shape-retaining fabrics like Red Jet Set. The experimental framework divided scenarios into two categories:

  • Training Scenarios: Lifting, Wind Blowing, and Stretching.
  • Evaluation (Unseen) Scenarios: Folding, Flinging, and Shaking.

Data was captured using a Baxter robot and ZED2i cameras, providing synchronized RGB-D images, point clouds, and precise robot motion trajectories.

Key Insights

The Impact of Simulation Engines

The choice of simulation engine proved to be as critical as the estimation method itself. The study found that DiffTaichi, which uses a simpler isotropic material model, struggled to capture the directional deformations inherent in real fabric. This often resulted in simulated cloth appearing overly rigid or exhibiting unrealistic "elastic-like" recovery during manipulation.

Scenario Efficacy for Parameter Estimation

Not all training scenarios are created equal. The "Lifting" scenario provided significantly more reliable data for estimating stiffness than "Wind Blowing." Lifting applies concentrated, well-defined tensile forces, whereas wind creates dispersed and fluctuating forces that are significantly harder to model accurately within current simulation constraints.

PINNs vs. Dynamic Complexity

The novel PINN approach emerged as the clear leader for quasi-static tasks. In the Folding evaluation, the PINN achieved an average error reduction of 16.5% compared to DIFFCLOUD, the next best-performing method. By solving underlying physics equations rather than just matching visual point clouds, PINNs provide a more grounded estimation of material properties.

However, high-speed dynamic scenarios like Flinging and Shaking remain a hurdle for all methods. The rapid motion makes tracking specific points on the fabric difficult, and current 2D mesh representations struggle to account for the complex self-occlusions and aerodynamic effects present in high-speed maneuvers.

Conclusion and Future Work

While Real-to-Sim approaches—and PINNs in particular—show great promise in bridging the Sim-to-Real gap for quasi-static tasks like folding, capturing the full spectrum of dynamic fabric behavior remains an open challenge. The research highlights a trade-off between local physical accuracy and global geometric matching.

Future research will focus on extending the PINN framework to 3D scenarios to move beyond 2D mesh approximations. Additionally, there is a pressing need for more robust, automated point-tracking methods that can maintain correspondence during high-speed deformations and complex self-occlusions.

Summary updated

Abstract

This paper presents a rigorous evaluation of Real-to-Sim parameter estimation approaches for fabric manipulation in robotics. The study systematically assesses three state-of-the-art approaches, namely two differential pipelines and a data-driven approach. We also devise a novel physics-informed neural network approach for physics parameter estimation. These approaches are interfaced with two simulations across multiple Real-to-Sim scenarios (lifting, wind blowing, and stretching) for five different fabric types and evaluated on three unseen scenarios (folding, fling, and shaking). We found that the simulation engines and the choice of Real-to-Sim approaches significantly impact fabric manipulation performance in our evaluation scenarios. Moreover, PINN observes superior performance in quasi-static tasks but shows limitations in dynamic scenarios.
BibTeX
@inproceedings{ru2025can,
  author = {Yingdong Ru and Lipeng Zhuang and Zhuo He and Florent P. Audonnet and Gerardo Aragon-Camarasa},
  title = {{Can Real-to-Sim Approaches Capture Dynamic Fabric Behavior for Robotic Fabric Manipulation?}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=Ae9WpJ6UPy},
}
Figure