Randomizing the Closure, Not the Friction: Learned Contact Corrections for Tactile Sim-to-Real

Fast robot simulators that run thousands of copies in parallel on a GPU treat contact crudely. A soft fingertip touching an object becomes a few rigid contact points with a single friction coefficient. The stretching of the skin, partial slip at the edge of the contact patch, and the spatial pattern of shear force under the finger are all thrown away. The usual repair is domain randomization: perturb friction, stiffness and mass widely during training so the policy learns to survive whatever the real world turns out to be. That makes policies cautious, and in tactile manipulation the field has gone further and binarized touch into contact or no contact. Both moves discard the shear and slip signals a robot needs for delicate force control.

We propose to borrow a tool from climate and turbulence modeling, the neural subgrid closure. This is a small network that runs inside a coarse numerical solver and adds back the effect of physics too fine for the solver's grid. Here the coarse solver is a GPU rigid-body engine, and the "subgrid" physics is continuum contact mechanics, learned from finite element simulations of a soft tactile skin. The closure corrects the contact forces the engine applies and also outputs a dense map of surface force under the sensor, which becomes the policy's touch observation. The second ingredient is the core claim. Rather than randomizing scalar friction and stiffness independently, we randomize over an ensemble of closures, each fit to finite element data for a different plausible skin material and surface. The policy then sees only perturbations that are physically consistent with each other, and should be less conservative than one trained under uniform randomization while transferring just as well.

Core Claim

Our main hypothesis is that, in tactile tasks where success depends on regulating slip and shear, policies trained in a GPU simulator with a learned contact closure transfer to hardware better than policies trained with broad parameter randomization or binarized touch. We expect this advantage to grow as the task demands more marginal frictional control. A secondary, cheaper hypothesis is that the policy reads the resulting dense touch history better when it is arranged as a structured pose-plus-motion image, as is done for skeletons in action recognition, than when it is given a raw stack of frames.

To make the closure concrete, let $s$ be the coarse simulator state near a contact (relative pose and velocity of sensor and object, penetration, and the engine's own normal and tangential force estimates). Let $F_{\text{FEM}}$ be the per-cell traction field on a $20\times 20\times 3$ grid produced by a finite element model, and let $F_{\text{eng}}(s)$ be what the rigid engine implies when spread over the same grid. The closure $g_\phi$ learns the residual

$$F_{\text{FEM}} \approx F_{\text{eng}}(s) + g_\phi(s, h),$$

where $h$ is a short history of past closure outputs, which carries hysteresis and stick-slip memory. The engine applies the summed residual force and torque back to the bodies, and the full corrected field is emitted as the tactile observation. This mirrors the residual definition $\tau = \overline{\mathcal{N}(u)} - \mathcal{N}(\bar u)$ used in fluid closures, with the rigid engine playing the role of the coarse operator.

Relation to Prior Work

Three lines of work motivate the proposal. In fluids and climate, Learned Simulators for Turbulence shows that a dilated convolutional network on coarse grids restores fine-scale structure that coarse solvers smear out, and that injecting noise into training inputs is needed to keep long rollouts from diverging. NeuralGCM couples a differentiable dynamical core with a learned tendency network and beats operational ensembles at much coarser resolution. Artificial neural network subgrid models of 2D compressible MHD turbulence shows that networks predict unresolved stress tensors better than classical gradient closures. We reuse their residual-learning interface and the noise trick. We take as a warning NeuralGCM's failure to extrapolate to warmer climates and the transfer-learning analyses (Explaining the physics of transfer learning a data-driven subgrid-scale closure): a closure is only trustworthy inside its training regime. This is why we randomize over closures fit to a spread of materials rather than trusting one.

The tactile work supplies both data and the target to beat. Sim-to-real for high-resolution optical tactile sensing generates Abaqus finite element trajectories and trains a network that maps marker images to 3D traction fields on real hardware. Its finite element model, however, is an offline data generator, not a component inside a dynamics simulator. Zero-Shot Sim-to-Real Transfer of Tactile Control Policies for Aggressive Swing-Up Manipulation does put a custom finite element contact model with stick-slip inside the training loop and distills a privileged expert into a touch-only student. That model is planar, runs at 360 Hz on a CPU, and cannot be replicated across thousands of GPU environments. Our closure aims to keep its physics at GPU throughput. VTDexManip is the clearest statement of the opposite strategy: binarize touch to cross the reality gap. Its authors concede that this rules out slip regulation, and that concession defines our target task family.

The simulator side constrains the design. Isaac Gym and MuJoCo Playground show that rigid engines plus aggressive randomization transfer well for grasping and locomotion. Any closure must therefore cost little enough that it does not erase their throughput, and it must win on tasks where randomization is actually the bottleneck. Intersection-free Robot Manipulation with Soft-Rigid Coupled Incremental Potential Contact argues that accurate contact can replace randomization. We test a learned middle ground between accurate but slow contact and fast but crude contact.

From action recognition, the touch-history encoding comes from Skeletal Movement to Color Map. That work lays out per-joint pose features and frame-to-frame motion features as an image indexed by joint and time, then classifies it with a standard 2D convolutional network. PointMapNet uses separate appearance and motion maps and reports that its motion map captures speed cues poorly. Following that report, our map carries explicit rate-of-change channels. Both papers do offline classification. We use the same layout for a closed-loop control input, and we treat it as an ablated add-on rather than a pillar of the proposal.

Research Program

The first stage builds the training data. We would rebuild an Abaqus-style finite element model of an optical tactile skin like that of Sferrazza and D'Andrea and extend it from quasi-static indentation to sliding, rolling and rate-dependent loading. Indenters and pole-like objects would be driven along trajectories sampled from what an RL policy actually does. Each trajectory is run for a grid of skin moduli, viscoelastic relaxation times and surface friction laws that brackets the hardware. In parallel, the same trajectories are replayed in MJX (MuJoCo compiled in JAX) to record $F_{\text{eng}}(s)$, which gives paired residual targets.

The second stage trains the closures. A small convolutional network on the $20\times 20$ sensor grid, with a recurrent state for hysteresis, is trained on single-step residuals with input noise as in Learned Simulators for Turbulence. It is then fine-tuned on short unrolled rollouts through the differentiable engine, in the spirit of NeuralGCM's online training. To protect conservation, the net normal force is constrained to be non-negative, the summed tangential force is clipped to a learned friction cone, and closure work is constrained to be dissipative during sliding. We would check energy drift directly rather than assume these constraints are enough, since soft constraints were insufficient in the turbulence work. One closure is trained per material setting, giving an ensemble of perhaps eight to sixteen members.

The third stage trains policies. Following Bi et al., a soft actor-critic expert sees privileged state, and a student that sees only the corrected traction history is distilled with DAgger (which repeatedly queries the expert on states the student visits). At each episode reset, one closure member is drawn, so randomization ranges over physically coherent contact behaviors. Ordinary randomization of mass and latency is kept, because the closure does not address those gaps. The student's input is either a raw 12-frame traction stack or the pose-motion map, which lays out cells by time with traction, its time derivative and its second derivative.

Baselines

The naive reference is a student trained in the plain rigid engine with only nominal parameters and no randomization. It gives the floor and tells us how large the contact gap is on each task.

The strongest competitor on the swing-up question is the custom finite element simulator approach of Bi et al.. It reports a mean absolute angle error of 4.3° from vertical across 40 trials on four poles. We would reimplement that pipeline on our rig, with the same expert algorithm, DAgger budget and observation window, so we compare against our own reproduction rather than a number measured on different hardware. Beating it means matching its angle error within trial-to-trial noise while training on a GPU at least an order of magnitude faster. On a slip-sensitive task where its planar model does not apply, beating it means a higher success rate.

The strongest generic competitor is broad parameter randomization in the style of MuJoCo Playground. This is the plain rigid engine with friction, stiffness, damping, mass and latency sampled independently. Touch is either computed from the engine's contact points or binarized as in VTDexManip. For fairness, its randomization ranges are tuned with the same number of hardware-free tuning runs we give our method, and both get identical environment-step budgets. Beating it means higher real-world success on slip-sensitive tasks and lower energy or lower grip force at equal success; the latter is the signature of reduced conservatism. Accuracy of the closure itself is compared against the real-hardware traction error that Sferrazza and D'Andrea report, an RMSE of 0.005 to 0.014 N per component. We found no reported result on Lacuna for any learned contact closure embedded in a GPU rigid-body engine, so there is no direct published number for that comparison.

Two ablations isolate the key ingredients. In the first, a single closure fit to the nominal material is combined with conventional scalar randomization layered on top. This tests whether ensemble-over-closures randomization matters beyond simply having a closure. In the second, the closure only shapes the touch observation and its force corrections are not fed back into the dynamics. This tests whether correcting the dynamics, rather than just the sensing, is what transfers.

Why not just…?

Why not just randomize friction and stiffness more widely? This is the real threat. On grasping and locomotion it demonstrably works, as MuJoCo Playground and Sim-to-Real: Learning Agile Locomotion For Quadruped Robots show. The known cost is over-conservatism. A rigid engine also cannot produce the spatial shear pattern that signals incipient slip, however its scalars are randomized. We would run randomization widths from narrow to broad and plot real success against grip force. If some width matches our method on the slip-sensitive tasks, the claim fails and we will report it.

Why not just binarize touch, as VTDexManip does? VTDexManip shows binarization transfers for cap turning and faucet screwing. Its authors also state that binarization rules out slip recovery and force regulation. We include binary touch as a baseline condition. We expect it to tie on contact-gated tasks and lose on tasks that require holding an object near the slip limit. If it does not lose there, dense touch is not worth its cost.

Why not just train a sensor-image-to-force network and leave the dynamics alone? This is the Sferrazza and D'Andrea recipe applied to observation only. It fixes what the policy feels but not how the object moves in simulation, so the policy may learn to exploit wrong slip dynamics. The second ablation, which corrects the observation only, tests exactly this. If it matches the full method, the dynamics correction is unnecessary and the proposal reduces to a better tactile renderer.

Why not just run the finite element model in the loop? On a small planar task, Bi et al. did, and it worked. It does not scale to 3D, multi-finger contact or thousands of parallel environments. We report wall-clock training time alongside accuracy so the trade-off is visible.

Proposed Experiments

Offline, closure accuracy is measured as per-component traction RMSE against held-out finite element trajectories, including material settings outside the training grid, so we can see where the closure breaks. Rollout stability is measured as energy drift and divergence rate over long unrolls, with and without the noise trick. Simulator throughput is reported with and without the closure.

In simulation, we use the pole swing-up task with poles of varied mass distribution, plus a tactile slip-hold task. In the slip-hold task, a gripper holds a sliding pole or rod at the minimum normal force that prevents a drop under a disturbance profile. The comparisons are against the baselines above, with the finite element simulator as held-out test physics, so that sim-to-sim transfer gives a cheap first verdict. The VTDexManip Shadow Hand tasks in Isaac Gym serve as a check on whether dense corrected touch beats binary touch in multi-finger settings, though only in simulation.

On hardware, which assumes access to a swing-up rig with an optical tactile gripper comparable to Bi et al.'s, we run zero-shot swing-up over 40 trials on four poles, matching their protocol, and slip-hold over a fixed set of objects. We report angle error, success rate and grip force. The decisive check has three parts: does the ensemble-closure policy achieve equal or better success than the best-tuned broad randomization policy on slip-hold, at lower grip force; does the observation-only ablation fall short of it; and does the pose-motion map beat the raw frame stack? If the first fails, the proposal's central claim is wrong. If only the third fails, we drop the action-recognition encoding and keep the closure.