Learning Fast, Non-Local Sim-to-Real Corrections for Deformable Objects

When a robot presses on soft tissue or pulls on cloth, its simulator predicts a shape, and the real object ends up somewhere slightly different. For deformable objects that difference is rarely confined to where the gripper touches. A stiffness error in one region, or a contact that grips a little harder than modeled, changes where every point on the object comes to rest. The best current fix recovers that correction by running an optimization loop online. That loop brings the error down to around a millimeter, but it takes one to three seconds per control step, which rules out closed-loop control. We propose to train a network that outputs the correction in a single forward pass. The network is built on one physical fact: the correction behaves like the solution of a boundary-value problem. Where and how the object is being touched, and which of its surface points can be seen, determine the correction at every interior point, including points far from the gripper.

The core claim is that a correction network made only of local message passing cannot represent these corrections once the error source and the affected region are separated by more hops than the network has layers. Local message passing is the standard graph neural network, in which each mesh node updates itself from its immediate neighbors. A network that sends the contact and observation boundary directly to every interior node can represent them, and should approach the accuracy of online optimization at more than ten times its speed. The claim is testable in both directions. We will also measure how far the real correction actually reaches across the object. If real sim-to-real corrections turn out to be short-range or low-dimensional, the long-range machinery is unnecessary, and the experiments are designed to show that too.

What exists and what is missing

The starting point is Liang et al.'s real-to-sim residual work. It couples a Position-Based Dynamics simulator with an online estimate of a per-node residual displacement and spatially varying stiffness. Position-Based Dynamics is a fast simulator that enforces geometric constraints between mesh nodes. The residual and stiffness are fit to real da Vinci Research Kit point clouds of chicken muscle using Chamfer distance, a nearest-point distance between two point sets. On volumetric tissue, average Chamfer error falls from roughly 4.2–5.7 mm to roughly 1.78–1.93 mm. On thin shells it falls from roughly 1.8–2.5 mm to 0.001–1.31 mm. The cost is 2.6 s per step for volumetric meshes and 0.9 s for thin shells, plus 0.4 s of perception. This paper supplies our training objective and our strongest baseline. We keep its idea that the sim-to-real gap can be written as a correction field on the mesh. We replace per-step optimization with a learned, amortized predictor.

The architecture comes from work on long-range propagation in physical graphs. BENO solves elliptic partial differential equations, a class in which every interior point depends on the whole boundary. It encodes the boundary values with a Transformer and injects that encoding into every interior message-passing layer, mimicking a Green's function decomposition. It improves on FNO, GKN and MP-PDE by an average of 60.96% on its corner datasets, where those baselines have relative L2 errors near 1.0. BENO was tested only on clean simulated PDEs with fixed boundaries. Our boundaries are instead the gripper contact set and the visible part of a noisy real point cloud, and both change from step to step. HCMT takes a different route to long range. It separates contact edges from internal mesh edges and routes stress through a precomputed coarse hierarchy, cutting position error by 49.2% over MeshGraphNet on Impact Plate. It too was never tested against real data. MiSe-GNN adds long-range edges where a learned error head predicts that local propagation fails. It needs two forward passes and depends on that head being well calibrated, which is a weak foundation under real sensor noise. All three papers motivate the proposal: they show that local propagation fails on physically global problems. None of them measures a sim-to-real gap.

One paper constrains the design. Kelesis et al. show that residual (skip) connections in deep graph networks such as GCNII and APPNP collapse them into weighted sums of shallow local filters. On the Star task, where the label depends on nodes 10 or more hops away, these models reach only about 65% accuracy, no better than chance. This bears directly on sim-to-real practice. The common recipe of "simulator output plus a learned local residual" silently caps how far a correction can travel, even if depth is added. Our model keeps the residual in output space, meaning the final state is the simulator's prediction plus a correction. It does not rely on layer-wise skip connections to reach far nodes. The long-range path is explicit.

The closest intersection work in the file does something else. SoMA learns a whole neural simulator from multi-view video over a hierarchical splat graph, rather than correcting an existing physics model. SSFold and FabricFlowNet sidestep the simulator for cloth policies. They do this with real demonstrations and with depth-only optical flow, respectively. None of these learns a fast correction to a physics model that is conditioned on the contact boundary.

Core Claim and Model

Let $x_{sim}$ be the node positions predicted by the nominal simulator after an action. Let $B$ be a set of boundary tokens: the gripper contact nodes with their commanded displacements, plus the observed partial point cloud $P$ encoded as surface patches. We learn a correction $r_\theta(x_{sim}, B)$ and train it with Liang et al.'s objective, applied without per-step optimization:

$$\mathcal{L}(\theta) = \mathrm{CD}\big(x_{sim} + r_\theta,\; P\big) + \lambda\, C_{PBD}\big(x_{sim} + r_\theta\big),$$

Here CD is the one-sided Chamfer distance from the observed points to the corrected mesh surface, $C_{PBD}$ is the summed violation of the simulator's geometric constraints, and $\lambda$ weights physical plausibility. The network has two parts, following BENO's split. A local graph network operates on mesh edges and handles short-range geometry. A small Transformer over the boundary tokens produces an encoding that is fed, through cross-attention, into every local layer. Information from a contact at one end of the object therefore reaches the far end in one step. We are not claiming that cross-attention to a boundary set is new. What is new is treating the sim-to-real correction itself as the unknown in a boundary-value problem. The contact and observation boundary change each step. The model is amortized so it can run inside a control loop.

Research Program

Real paired data are scarce, so training runs in two stages. The first stage uses a synthetic reality gap. "Reality" is a finite-element simulation with randomly drawn heterogeneous stiffness fields and contact friction, and "sim" is nominal Position-Based Dynamics. This yields unlimited pairs with known ground-truth corrections. It also gives a knob that no real dataset provides: we can place the stiffness error at a chosen hop distance $d$ from the gripper, and vary mesh size so the diameter exceeds the network depth. The second stage fine-tunes and evaluates on the real dVRK chicken tissue data from Liang et al., both thin-shell and volumetric, using the loss above against the observed point clouds. We then test transfer to a second real domain, the ARX-Lift and PhysTwin multi-view rope, doll and cloth recordings used by SoMA. These recordings let us check whether the approach extends beyond tissue to thin, high-diameter objects.

Alongside the method, we run a measurement study that any honest version of this claim requires. We take the corrections Liang et al.'s optimizer produces offline on the real tissue sequences. We then compute two things: their spatial correlation length relative to mesh diameter, and how many principal components explain 95% of their variance. These two numbers tell us, before any architecture comparison, whether real sim-to-real corrections are long-range, and whether they are low-dimensional.

Baselines

The naive reference point is the uncorrected nominal simulator. On the real tissue data this corresponds to Liang et al.'s pre-correction error of roughly 4.2–5.7 mm Chamfer on volumetric tissue and 1.8–2.5 mm on thin shells. Any learned correction must clearly beat this.

The strongest existing method is Liang et al.'s online residual and stiffness optimization, run on the same sequences with its reported settings. It reaches roughly 1.78–1.93 mm volumetric Chamfer at 2.6 s per step, and roughly 0.001–1.31 mm thin-shell Chamfer at 0.9 s per step. Its speed-accuracy trade-off can be adjusted by capping iterations, so we will plot its full error-versus-latency curve rather than a single point. Beating it means one of two things. Either our model sits within 0.2 mm Chamfer of the converged optimizer at under 50 ms per step, or it lies below the optimizer's curve at every matched latency. We will also report whether our prediction used as a warm start shortens the optimizer's run.

The ablation that isolates the key ingredient is the same model with the boundary cross-attention removed. What remains is a purely local graph network with matched parameter count, depth and training budget. We will build it in two variants: plain stacked layers, and GCNII-style skip connections, which directly tests Kelesis et al.'s anchoring effect on correction fields. We add two further long-range comparisons, each with the same data and tuning budget. The first adds a single global virtual node connected to every mesh node, the cheap shortcut used in Graphormer and studied by Manolache et al.. The second is a GPS-style hybrid, GPS, which pairs local message passing with linear global attention over all nodes but is not told which nodes form the boundary. GPS is reported to improve F1 on COCO-SP by about 25% over pure message-passing networks, making it the strongest generic long-range competitor. If GPS matches our model, the boundary structure adds nothing beyond global mixing.

Every learned model gets the same synthetic pretraining set, the same real fine-tuning split, and the same number of hyperparameter trials, and each is chosen by validation Chamfer. We report latency on identical hardware.

Why not just…?

Why not just stack more message-passing layers with skip connections so the network covers the object? Kelesis et al. show that skip-connected deep graph networks behave like shallow ones and fall to about 65% accuracy on the 10-hop Star task. Without skip connections, deep stacks run into the oversquashing that GPS and BENO were designed to avoid. Our distance-controlled synthetic experiment tests this directly for correction fields. It measures how error grows with the hop distance between stiffness error and gripper, for both the plain and skip-connected local variants.

Why not just randomize stiffness and friction in simulation and skip the correction model? Randomization makes a policy robust to a spread of parameter values. It does not tell the robot where this particular piece of tissue actually is. The failure cases in the file for deformables, such as SoftGym's FleX physics missing stick-slip and multi-layer contact (Lin et al.), are structural model errors, not parameter spread. The nominal-simulator baseline, and a variant trained with randomized stiffness, will show how much of the real error randomization alone removes.

Why not just fit a low-dimensional correction, a few stiffness regions or a handful of deformation modes, and solve for its coefficients? This is the real threat. Elliptic-like corrections are smooth, and smooth fields are often well captured by a few modes. We will build this baseline properly. It uses a principal-component basis of corrections from the synthetic set, with coefficients predicted by a small network from the same boundary tokens. If it matches our model on real tissue, the right conclusion is that sim-to-real corrections are low-rank and a mesh-wide long-range architecture is overkill. The measurement study predicts in advance which outcome to expect. We expect the modal basis to break down when the contact location moves or the stiffness varies across the object, because the dominant modes then change. The distance-controlled and contact-shift splits test exactly that.

Why not just warm-start Liang et al.'s optimizer and run fewer iterations? The matched-latency curve answers this. If a truncated optimizer at 50 ms equals our model, amortization buys nothing and we will say so.

Proposed Experiments

Three results decide the question. The first is the synthetic distance sweep. We predict that the local and skip-connected variants degrade sharply once the hop distance $d$ exceeds their layer count, while the boundary-conditioned model and GPS stay flat. The virtual-node model should sit in between, because a single vector must summarize the whole boundary. The second is the real dVRK comparison, reported as Chamfer error on volumetric and thin-shell tissue against the uncorrected simulator and against Liang et al.'s error-versus-latency curve. The target is the converged optimizer's accuracy, within 0.2 mm, at under 50 ms per step. The third is the correlation-length and rank measurement on the real optimized corrections. It tells us whether any gain we see comes from long-range structure or from something a low-rank model would also capture.

As a secondary check of generality, we will fine-tune on the ARX-Lift and PhysTwin sequences and report geometric error relative to SoMA, which is reported at 33.51 PSNR on resimulation and about 20% better than PhysTwin and GausSim. This comparison is only indicative, since SoMA learns a whole simulator and we correct an existing one. The proposal succeeds if the boundary-conditioned model wins on the distance sweep, stays within 0.2 mm Chamfer of the online optimizer at over ten times its speed, and beats the low-rank modal baseline on contact-shift splits. If the modal baseline ties, the paper becomes a measurement result: sim-to-real corrections for soft tissue are low-dimensional, and that is worth knowing too.