Translating Real Touch Back to Simulation Needs Memory

A soft robotic fingertip remembers. Its rubber skin creeps under steady pressure and recovers slowly when released. On the way down and on the way up, it deforms differently for the same contact, a lag called hysteresis. Fast tactile simulators have no such memory: they render each instant from the current contact alone. A popular way to deploy simulation-trained touch policies is to translate each real camera frame from inside the fingertip into the clean image the simulator would have produced. That translation is done one frame at a time, so it tries to undo a sensor with memory using a function without memory. We argue this is a structural reason, not just a modeling detail, why such transfer jitters and breaks on dynamic tasks. We propose a real-to-sim translator that is recurrent, and that is told by the robot's own joint readings whether the skin is being loaded or unloaded. It outputs two separated channels, normal indentation and tangential shear, that a cheap simulator can also render.

The concrete test is the task suite where this approach is known to fail. Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation trained policies on rigid-body depth maps in PyBullet and translated real TacTip images frame by frame with an image-to-image GAN. It transferred edge following, surface following, pushing and rolling, but Object Balancing failed. The authors attribute this to missing shear and to latency, and they note jitter under fast motion. Our claim is that adding shear to the simulator is not enough. Shear and depth can only be recovered from a viscoelastic skin if the translator also sees the contact history and the direction of motion.

Core Claim

Write the real sensor image as $o_t = g(s_t, m_t)$. Here $s_t$ is the instantaneous contact state the simulator models (normal indentation field and tangential displacement field), and $m_t$ is the skin's internal material state, which evolves as

$$m_t = f(m_{t-1}, s_t).$$

When hysteresis is present, the same image $o_t$ arises from different $s_t$ depending on $m_t$. No function of $o_t$ alone can then recover $s_t$, and a frame-wise translator must average over the loading and unloading branches. That averaging appears as a phase-dependent bias. A translator conditioned on the image history and on proprioception can estimate $m_t$ and remove the ambiguity. Proprioception here means joint encoder readings at hundreds of hertz, which directly report indentation velocity and lateral motion.

This yields a prediction that frame-wise methods cannot fake. Translation error should split by loading phase and loading rate, and the split should vanish with history and proprioception.

The four research directions play distinct roles. Signal decoupling defines the output: separate normal and shear channels instead of one entangled depth image. Spatio-temporal modeling is the mechanism: a small convolutional recurrent network that keeps the taxel layout inside its memory, as ConvLSTMs do in Short-Term Traffic State Prediction Based on the Spatiotemporal Features of Critical Road Sections. Multimodal fusion is narrow and purposeful: proprioception enters the translator as a loading-direction cue, not as extra policy input. The sim-to-real noise gap is the evaluation regime.

Relation to Prior Work

Two works supply the method we build on. Church et al. supply the real-to-sim translation setup and paired data collection: the robot replays poses in sim and reality, so real images get exact simulated targets. Their translator is memoryless and their target lacks shear. Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions shows that a light U-Net can decouple $F_x, F_y, F_z$ fields from a single marker image at 120 Hz. It is trained on Abaqus finite-element data, operates quasi-statically, and its authors name hysteresis as future work. We keep the decoupled output but drop the expensive simulator and add memory.

Three works motivate the gap.

Two results show that history helps perception and transfer. Grasp Stability Prediction with Sim-to-Real Transfer from Tactile Sensing shows a ResNet-18 plus LSTM over tactile sequences transferring to real grasps. Seeing-Eye Quadruped Navigation with Force Responsive Locomotion Control shows a 1D convolution over 25 proprioceptive steps separating a human tug from gait noise. Neither places memory in the observation translator, where the simulator's memorylessness actually bites. Si et al. also put temporal modeling in the downstream classifier, which must then be trained on simulated data that never contained hysteresis.

One work constrains the design. Church et al. argue that "geometry often trumps physics." Si et al. found the opposite for grasp stability, where a geometric simulator reached 42.6% accuracy against 90.7% for a physics-calibrated one. We take a middle position. The simulator stays geometric and fast; we add only a kinematic stick-slip shear channel. The translator absorbs the physics the simulator omits.

We do not address vision–touch decoupling inside a shared camera, as in ViTacTip. The same history argument would apply to its marker-removal GANs, which also run frame by frame, but that is a later extension.

Research Program

Simulator. We extend the Tactile-Gym PyBullet environment with a second rendered channel. At each contact point, it computes the tangential offset of the sensor from where that point first touched the object. The offset is clipped when it exceeds a Coulomb friction bound, and the stick reference resets when slip occurs. This keeps rendering near the simulator's original speed while giving policies a shear signal. PPO policies are retrained on depth plus shear for all five tasks, including Object Balancing.

Paired data. We collect real TacTip sequences on a UR-series arm under scripted indentation, shear and slide trajectories. Indentation rates and dwell times are varied deliberately so that the data contain creep, relaxation and both hysteresis branches. Each trajectory is replayed in simulation to obtain canonical depth and shear targets.

Translator. The translator is a convolutional GRU running on downsampled marker images at camera rate. At each step it receives a feature computed from the joint states recorded since the previous frame, which capture indentation and lateral velocity. Training uses an L1 loss on both output channels plus an optional adversarial term. Model size is matched to the Church translator.

Analysis tool. The main analysis tool is the hysteresis loop test. Flat and edged indenters are pressed and released at several rates, and translation error is plotted separately for loading and unloading.

Baselines

Naive reference. The do-nothing reference is the published pipeline of Church et al.: a frame-wise Pix2Pix translator producing depth only, driving depth-only PPO policies. Its reported real-world results are a mean edge-following distance error of about 1.5 mm and a surface-following depth error of 0.57 mm. Object Balancing is reported only as failed; no numeric success rate was found on Lacuna. We reproduce this baseline on our own hardware before comparing against it.

Strongest frame-wise competitor. This baseline is the same frame-wise translator retrained to output our depth and shear channels, driving the shear-aware policies. It isolates whether simply adding shear to simulation fixes balancing; if it does, our temporal argument is unnecessary. We also run a Sferrazza-style U-Net producing decoupled force channels per frame, followed by a tuned low-pass filter. This is the cheapest way to suppress jitter. Its published RMSE of 0.005–0.014 N per force component was measured on a different sensor and is not directly comparable.

Frame stacking. A frame-wise translator receives the last $k$ frames, with $k \in \{4, 16, 64\}$. This gives it history without recurrence or proprioception.

Ablations. These isolate the key ingredients:

  • our recurrent translator without proprioceptive input;
  • proprioception with a single image frame;
  • the full model trained on depth targets only.

Fairness and success criterion. All translators use the same paired dataset, a parameter budget within 10% of each other, and the same hyperparameter search budget of twenty configurations each, selected on a held-out set of real trajectories. All drive identical frozen policies. Beating the strongest baseline means three things: lower translation error on the unloading branch and at the fastest indentation rates, measured against robot kinematics and a wrist force/torque sensor; no loss on the slowest, quasi-static trajectories; and a nonzero, repeatable Object Balancing success rate where the shear-only frame-wise variant stays near zero.

Why not just…?

Why not just stack recent frames into the existing translator? This is the most serious alternative, and the frame-stacking baseline exists to test it. A short stack captures velocity but not creep over a multi-second dwell. Even a long stack cannot tell loading from unloading when the skin is momentarily static at the turning point, while proprioception reports the direction directly. If a 16- or 64-frame stack matches the full model on the hysteresis loop test and on balancing, we will report that "history, not recurrence" is the finding. In that case the contribution becomes the shear-aware simulator and the decoupled target, and the recurrent architecture would be dropped.

Why not just randomize hysteresis in simulation so the policy learns to tolerate it? Randomization made pin-based perception robust to physics variation in Sim-to-Real Transfer for Optical Tactile Sensing. Binarizing and noise injection helped resistive arrays in Sim-to-Real Transfer for Robotic Manipulation with Tactile Sensory. Randomizing a viscoelastic filter on simulated channels, however, teaches the policy to tolerate a biased shear estimate rather than to remove the bias. For balancing, which depends on small shear differences, tolerance means sluggishness. We will test this directly by training a policy with a randomized first-order viscoelastic filter on the simulated channels, paired with the frame-wise translator. If it balances as well as our pipeline, the translator's memory is not needed.

Why not just fine-tune the policy on real data? Fine-tuning gives up zero-shot transfer and needs real rollouts on a balancing task that drops the object on failure. The translator needs only scripted, self-supervised indentation data, which is collected once per sensor. We will report the number of real fine-tuning episodes a frame-wise pipeline needs to match zero-shot balancing, as an exchange rate.

Why not decouple normal and shear in hardware? Sensors like GTac separate normal and shear components physically. They do not remove the elastomer's memory, and they require new hardware. Our question concerns existing low-cost optical skins, which ViTacTip argues are worth keeping simple.

Decisive Checks

The first check is the hysteresis loop test. Frame-wise translators should show a systematic gap between their loading and unloading errors that widens with indentation rate. Our model should close that gap, and removing proprioception should reopen it mainly at turning points. If the frame-wise gap is absent on our TacTip, the premise fails and we stop.

The second check is closed-loop. All variants are run on the five Tactile-Gym tasks on real hardware, with ten trials per task per variant. The first four tasks should match Church et al.'s reported errors. Object Balancing is the discriminating task.

The third check measures latency of the full chain against camera rate. The recurrent translator must run within one frame; otherwise we have only traded hysteresis lag for compute lag.