Single-View RGB-D Force--Deformation Estimation for Robotic Manipulation of Elasto-Plastic Materials

Dezhi Sun, Marijn van Hilten, Kenan NiuarXiv 2026 · arXiv

When a robot pushes into a lump of clay or dough, the material doesn't just move—it changes. For "elasto-plastic" materials, these changes are persistent, meaning the robot’s interaction history is literally written into the object's shape.

In their paper, "Single-View RGB-D Force–Deformation Estimation for Robotic Manipulation of Elasto-Plastic Materials," Dezhi Sun and colleagues from the University of Twente present a practical, minimal-sensing framework to bridge the gap between what a robot feels (force) and what it sees (deformation).

TL;DR

The researchers developed a system using a standard 7-DoF robot and a single eye-in-hand RGB-D camera to estimate the relationship between interaction force and persistent surface deformation. By mapping displaced volume to intrinsic force estimates, they established a measurement basis for "deformation-aware" manipulation without needing specialized tactile sensors or complex multi-camera rigs.

What Problem Does This Paper Address?

Manipulating deformable objects is a "holy grail" in robotics because every touch alters the object's geometry and future dynamics. Elasto-plastic materials (like Play-Doh or clay) are particularly tricky because they don't spring back to their original shape.

To shape these materials intelligently, a robot needs to know: "If I press with $X$ Newtons of force, how much will the surface change?" Most existing research focuses on high-level goal geometry or complex particle simulations. This paper zooms in on the fundamental physical link: the force–deformation relationship, measured using commodity hardware already found in many robotic workcells.

Methods

The authors used a Franka Emika FR3 robot equipped with an Intel RealSense L515 camera. Crucially, they relied on the robot's intrinsic force estimation (derived from joint torques) rather than an external force/torque sensor.

The workflow follows a staged approach:

  1. Pre-contact scan: The camera captures the initial surface.
  2. Controlled Indentation: The robot performs a normal press until a target force is reached.
  3. Post-contact scan: After the tool withdraws, the camera captures the persistent dent.
  4. Volume Estimation: A point-cloud pipeline (using Open3D) registers the two scans, fits a reference plane using RANSAC, and calculates the estimated displaced volume ($V$).
The system setup and the point-cloud processing pipeline for volume estimation.

The team compared two modeling approaches to map volume back to force:

  • Analytical Baseline: A simple quadratic model.
  • Learning-Based: A compact Multilayer Perceptron (MLP) with a 1–32–32–1 architecture.

Datasets

The study utilized several small, targeted datasets collected on a Play-Doh-type material:

  • Characterization Set: 27 measurements across nine force levels (5N to 25N) to establish the general trend.
  • MLP Training/Test Set: 52 force–volume pairs at four specific levels (15, 17.5, 20, 22.5 N).
  • Comparison Set: 9 pairs at three intermediate force levels (16, 19, 23.5 N) to evaluate model generalization.
  • Sequential Set: An 8-step experiment applying adjacent indentations to test cumulative deformation tracking.

Key Insights and Ideas

  • The "Force Floor": The researchers found that below ~10N, surface displacement is difficult to distinguish from sensor noise and background variation. This defines an "experimentally useful" range for this specific hardware setup.
  • Trajectory Drift: Even with identical force targets, the robot's physical path can vary by several millimeters (see Figure 2). This highlights why force-based termination is more robust than pure position control for shaping tasks.
  • Model Complementarity: While the MLP captured nonlinearities better at higher forces, the quadratic model was surprisingly effective and more interpretable in the mid-range. The authors argue against picking a "winner" based on a single aggregate score, as different models excel at different force regimes.
  • Cumulative Localization: Instead of trying to identify individual "dents," their sequential pipeline tracks a "dominant affected region." This allows the robot to follow the evolution of a shape as multiple interactions merge into one larger deformation.
Sequential tracking shows the dominant affected region expanding as multiple indentations are made.

Key Results

  • Force Estimation Accuracy: The MLP achieved a held-out RMSE of 1.857 N and an $R^2$ of 0.440.
  • Model Comparison: The quadratic model showed lower error at 16 N, while the MLP performed better at 19 N and 23.5 N.
  • Repeatability: The system successfully localized cumulative deformation from the second indentation onward in sequential trials, providing a stable centroid and volume estimate for planning.
Force-volume characterization (left) and MLP prediction performance (center).

Limitations and Future Work

The authors are transparent about the constraints of this "minimalist" approach:

  • Single-View Blindness: A single camera cannot see "self-occlusions" or material that is displaced laterally (the "bulge" around a dent).
  • Material Specificity: The results are currently limited to one type of material under controlled lab conditions.
  • Open-Loop: This work establishes the measurement foundation; the next step is integrating these models into a closed-loop controller for autonomous sculpting.

By proving that standard hardware can reliably link force to persistent shape change, this research offers a practical path forward for robots that need to do more than just move objects—they need to reshape them.

Summary updated

Abstract

Persistent deformation complicates elasto-plastic manipulation by coupling physical interaction with spatially distributed shape changes. We investigate the force--deformation relationship using a standard robot, a single eye-in-hand RGB-D camera, and intrinsic force estimates. A 7-DoF Franka Emika FR3 performs normal indentations while an Intel RealSense L515 captures pre- and post-contact surfaces. Point-cloud processing localizes deformation and estimates displaced volume. Force--volume characterization includes 27 measurements over nine force levels. A compact multilayer perceptron (MLP), trained on 52 force--volume pairs at four levels, achieves a held-out RMSE of 1.857 N and $R^2=0.440$. A separate comparison shows lower error for a quadratic model at 16 N and for the MLP at 19 and 23.5 N. An eight-step experiment at 4 N per interaction demonstrates cumulative deformation localization from the second indentation onward. These results establish a practical measurement basis for deformation-aware manipulation using intrinsic force estimates and single-view geometry.
BibTeX
@misc{sun2026single,
  author = {Dezhi Sun and Marijn van Hilten and Kenan Niu},
  title = {{Single-View RGB-D Force--Deformation Estimation for Robotic Manipulation of Elasto-Plastic Materials}},
  year = {2026},
  eprint = {2611.15650},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15650},
}
Figure