Robots that fold cloth or route cables are almost always trained in simulation and then moved to real hardware. The simulators get the physics of soft materials only partly right: fabric friction, bending stiffness and grasp slip are routinely off. So a policy that respected a safety limit in simulation, such as a cap on how hard it pulls on the fabric or presses on the table, can break that limit on the real robot without any warning. Our idea is to treat the reality gap as the same kind of problem offline reinforcement learning (RL, learning to act from trial-and-error reward) already knows how to handle. That problem is extrapolation error: a learned value estimate is trusted in places where no data supports it. Here the unsupported places are the state-action regions where the simulator mispredicts reality, not merely actions missing from a dataset. We propose to measure, from a small log of safe real-world motions, where and by how much the simulator is wrong. The learned estimate of future safety cost is then inflated only where two things hold at once: the simulator is uncertain or wrong, and the cost is sensitive to that error.
In two sentences for a colleague: a cost critic trained with domain randomization (training across many randomly perturbed simulator physics settings) underestimates danger precisely where the real physics falls outside what any simulator setting can reproduce. We learn that mismatch from a few real episodes and add a penalty to the critic that scales with the mismatch times the critic's sensitivity to it. Everywhere else the robot keeps the task performance of ordinary sim-trained policies.
Core Claim
Current safe sim-to-real methods split along two lines. Provably Safe Sim-to-Real Transfer estimates a "mismatch region" where simulator and reality disagree and concentrates real exploration there, reaching 0% unsafe iterations against 68.5% for an unconstrained baseline. It is tabular, though, and assumes the simulator is either exactly right or wrong by at least a fixed margin. Soft-body physics violates that assumption, since discrepancies in bending or friction vary smoothly with the cloth's configuration. Sim-to-Lab-to-Real scales safety shields to continuous vision-based control, but only for rigid navigation and with hundreds of intermediate training environments. Offline safe RL methods, C2IQL and DRCORL, fight cost underestimation on unseen actions using in-sample value updates or uncertainty-penalized cost ensembles. Both assume the training and test dynamics are identical.
Our claim is that for deformable manipulation, the dominant source of real-world constraint violation is cost underestimation along the dynamics axis. It arises from structural simulator misspecification that domain randomization cannot absorb. It can be corrected by a continuous, state-dependent pessimism term estimated from a small real log. We expect this correction to buy a better safety–performance trade-off than the obvious fix, which is to tighten the safety budget uniformly.
Mechanism
Let $Q_c(s,a)$ be the cost critic trained in the randomized simulator: the expected discounted safety cost (for example, time spent above a grasp-tension or contact-force threshold) from state $s$ after action $a$. Let $V_c(s')$ be the corresponding state-value estimate at the next state $s'$.
We start with a small real log, tens of episodes collected with a conservative scripted or demonstrated policy. From it we first fit the randomization distribution by likelihood, as in DROPO. That method's parameter variance soaks up some unmodeled effects, but Statistical Guarantees for Offline Domain Randomization shows it can only converge to the best projection within the simulator's model class. What remains is the residual $\delta(s,a)$ between the real next state and the best-fit simulator's prediction. We learn this residual with an ensemble of small models, each trained on the real log. For any query $(s,a)$ we take an upper-confidence bound $u(s,a)$, meaning the ensemble's predicted residual plus a multiple of its disagreement. In regions the real log never visited, the disagreement is large, so the bound falls back to a large prior value. The deployed critic is
$$Q_c^{+}(s,a) = Q_c(s,a) + \beta \, \mathbb{E}_{s'}\big[\,\|\nabla_{s'} V_c(s')\|\,\big]\; u(s,a),$$
where $\beta$ is a single scalar and the gradient norm measures how much the predicted future cost changes per unit error in the next state. The form comes from a first-order Taylor bound: if the true next state is within $u$ of the simulated one, the value error is at most roughly the gradient norm times $u$. The penalty therefore vanishes where the simulator is accurate, and also where it is inaccurate but the error is irrelevant to safety. An example of the second case is a mispredicted wrinkle far from any force limit.
Because the real log is collected safely, it rarely visits states near the constraint boundary. Feasible Dual Policy Iteration calls this the safety paradox. We handle it in two ways. The upper bound defaults high off-support. We also run one or two rounds of shielded real exploration. In each round, actions whose $Q_c^{+}$ exceeds the remaining budget are replaced by a backup primitive (release and retreat), borrowing the override logic of Sim-to-Lab-to-Real. Each round grows the region where $u$ is small. Following C2IQL, the policy receives the remaining budget as an input, so it can become more cautious as the budget is spent.
Research Program
We build on pick-and-place primitive actions rather than low-level torques, which behavior-primitive work shows insulates policies from timing mismatch. Critics use keypoint or graph representations of the cloth or rope rather than flattened particle vectors, because SoftGym found flat full-state MLPs perform poorly on deformables. The critic sees privileged simulator keypoints, as in the asymmetric actor-critic of Pinto et al. used by Matas et al. and DMfD. On real data, keypoints come from a tracker, and cost comes from a wrist force-torque sensor.
The main testbed is cross-simulator transfer, which gives controllable structural misspecification with ground-truth cost. The policy is trained with domain randomization in DaXBench, whose cloth is a mass-spring model and whose rope uses the material point method. It is then deployed in SoftGym, whose cloth and rope run on a different particle engine (FleX). The paired tasks are Fold-Cloth/Unfold-Cloth mapped to SoftGym FoldCloth/SpreadCloth, and Push-Rope mapped to StraightenRope. No DaXBench parameter setting reproduces FleX exactly, so this is misspecification by construction. We add a second, graded axis: hidden effects injected into the "real" simulator that the training family cannot express, such as anisotropic bending or a position-dependent friction patch. A final stage is a physical Franka cloth-folding setup mirroring DMfD, with a fabric-tension limit enforced through measured grasp force.
Baselines
The naive reference is a policy trained with uniform domain randomization over wide ranges and an ordinary Lagrangian constrained learner, deployed with no real data. DROPO reports that uniform randomization reaches about 10 cm pushing error against DROPO's roughly 2 cm. This baseline tells us how much simply ignoring the gap costs.
The strongest simple competitor is DROPO-fitted randomization with a FOCOPS learner, which maximizes reward subject to a cost budget using first-order updates. In FOCOPS it reports 1106.1 reward on Humanoid-Circle versus 671.0 for CPO, within the cost limit. We add a uniformly tightened budget, sweeping the margin to trace a full return-versus-violation curve. It uses the same real log, simulator steps and hyperparameter search budget as our method. Beating it means achieving higher task success at equal real-world violation rate across the curve, not at a single point.
The strongest offline-safe competitor is DRCORL, which reports a normalized return of 0.91 on DSRL CarGoal1 against 0.38 for the next safe baseline. We run it with its UCB cost ensemble on the union of simulator rollouts and the real log, with the same data and budget. We also run C2IQL the same way.
For deformable task success the reference numbers are 85.58% real success for DMfD and 66.6% real diagonal folding for Matas et al.. No reported result was found on Lacuna for constraint-violation rates on any deformable sim-to-real task, so the safety comparison is new ground and all violation numbers will come from our own runs.
The ablations isolate the key ingredient. Setting $u=0$ gives fitted randomization plus shield. Replacing the gradient-weighted term with a constant $\beta u$ tests whether sensitivity weighting matters. Replacing residual uncertainty with an ensemble over randomization parameters only tests whether in-family uncertainty suffices.
Why not just…?
Why not just tighten the cost budget by a constant margin? This is the real threat. If mismatch-induced violations are spread evenly across the state space, a constant margin is equivalent to our method and simpler. We predict they are concentrated, for example during lift-and-drag phases where friction errors dominate, so a constant margin wastes performance in easy phases. The margin sweep in the baselines settles this directly. If the curves coincide, we will report the negative result and the proposal reduces to the fitted-randomization baseline.
Why not just use a critic ensemble across randomized physics parameters? Such an ensemble measures disagreement among simulator settings, which is uncertainty within the model class. Statistical Guarantees for Offline Domain Randomization makes clear that misspecified reality lies outside that class. The graded hidden-effect experiments test this. If the parameter ensemble catches violations as well as residual uncertainty does even when the injected effects are inexpressible, our mechanism is unnecessary.
Why not just stop when the wrist force exceeds a threshold? A force stop is a useful last line of defense, and we keep it on hardware. However, Constraint-aware Policy for Compliant Manipulation reports that classical compliance controllers need task-specific tuning. A reactive stop also acts only after the force has risen, whereas tearing or crushing can happen within that window, and dynamic motions contaminate force readings with inertia. We report the force-stop-only baseline's violation magnitude as well as its frequency.
Why not just run offline safe RL on the real log? With tens of episodes on high-dimensional cloth states, the log is far too small. DMfD needed about 1,000 or more expert episodes even in simulation. The C2IQL-on-real-only run quantifies this.
Decisive Checks
The first check is diagnostic and precedes any method claim. In cross-simulator transfer, we test whether real violations of the plain randomized policy concentrate where the simulator residual and the cost gradient are both large. If they are uniformly distributed, the core hypothesis fails. The second check is the Pareto comparison against the tightened-budget FOCOPS baseline on the three DaXBench-to-SoftGym task pairs. We measure task success (SoftGym's normalized performance) against the fraction of episodes violating the tension or force limit, and against total real episodes consumed. The third check sweeps hidden-effect strength: our advantage over the parameter-ensemble ablation should grow with misspecification and vanish when the "real" simulator lies inside the training family. Only if all three hold do we move to the physical Franka trial. There we compare fold success and grasp-force violations against fitted randomization plus force stop under the same real-episode budget.
References
- Provably Safe Sim-to-Real Transfer
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning
- Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL
- DROPO: Sim-to-real transfer with offline domain randomization
- Statistical Guarantees for Offline Domain Randomization
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
- SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
- Asymmetric Actor Critic for Image-Based Robot Learning
- Sim-to-Real Reinforcement Learning for Deformable Object Manipulation.
- Learning Deformable Object Manipulation From Expert Demonstrations
- DaxBench: Benchmarking Deformable Object Manipulation with Differentiable Physics
- First Order Constrained Optimization in Policy Space
- Constraint-Aware Policy for Compliant Manipulation