Learned Contact Vocabularies for Tactile Sim-to-Real Transfer

Robot hands trained in simulation transfer to real hardware best when their touch sensors are reduced to almost nothing: each sensing element ("taxel") reports only "touching" or "not touching". The reduction works because a physics simulator can compute contact on/off reliably, while real sensor readings are distorted by gel mechanics, lighting, magnetic hysteresis and wear in ways no fast simulator reproduces. The cost is high. The robot cannot tell a light touch from a crushing one, and it cannot sense slip before the object moves. On top of that, the on/off cut is a hand-tuned threshold that drifts as the sensor ages. Speech translation faced a similar problem, where one modality is plentiful and the paired data is scarce. There, mapping both modalities onto a shared discrete vocabulary, anchored by a cheap bridge dataset, turned a failed transfer into a working one.

We propose to treat the binary threshold as what it already is, a two-word vocabulary shared between simulator and sensor. The work is to make that vocabulary learnable and larger without reopening the simulation gap. The words are defined only by quantities the simulator computes well: contact, a few coarse force levels, and shear direction. A small learned classifier on the real sensor predicts those words from raw readings. It is trained on a cheap bench calibration dataset, never on policy rollouts. The classifier's measured confusion is then injected back into simulation during training. The central empirical question is the shape of the trade-off between vocabulary size and transfer quality. We want to know how many contact words a policy can use before the simulator's inaccuracy in force magnitude starts costing more than the extra information gains.

Core Claim

Hard binarization is one point on a curve, not an optimum. A physics-defined contact alphabet of size $K>2$ yields better real-world performance than binary or hand-thresholded ternary codes on tasks that need force modulation or slip awareness, provided two conditions hold. First, the real-side encoder must be learned from calibration data rather than thresholded. Second, its errors must be modeled in simulation. The same learned encoder should also make transfer robust to sensor drift, because a few minutes of recalibration replaces it without retraining the policy. If the curve peaks at $K=2$, binary abstraction is vindicated as a principled choice rather than a heuristic. That would also be a useful result.

Grounding in Prior Work

Several papers motivate the gap. Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning compared raw DIGIT images, difference images and thresholded binary masks. The binary mask transferred best, reaching 80% real pivoting success on 16 unseen objects against 33% without touch. Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensing showed that binary normal contact was not enough for translation. It added hand-thresholded ternary shear on a ReSkin array and reached 93% success on out-of-distribution objects. VTDexManip and Wu et al. also binarize by fixed threshold. All four fix the vocabulary by hand. None asks whether a richer, learned vocabulary transfers, and none handles threshold drift. Our proposal keeps their simulator-side definition of contact and replaces their real-side threshold with a learned, recalibratable predictor over a larger alphabet.

One paper supplies the method. Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation maps speech and text into shared discrete codes, using paired speech–transcript data as the bridge. It reports 22.4 BLEU zero-shot on MuST-C En-De against about 6.8 for continuous alignment, and 1.3 when quantization is removed. We borrow its structure. A shared discrete vocabulary is anchored by a cheap bridge dataset, here bench calibration in place of speech recognition data, and the scarce target pairing is never collected, here real policy data in place of speech translation data. We differ in one important way. We do not learn the codebook end to end, because the simulator cannot render realistic raw tactile signals for an unsupervised codebook to align with. Instead we fix code meanings in simulator-computable physics. An unsupervised codebook is included as an ablation.

Other work constrains the design. The continuous visuo-tactile methods VITaL Pretraining, ViTacFormer, DECO and Tactile-VLA get force-level information only by collecting real teleoperation data. DECO, for example, uses 50 hours. This shows that continuous touch is valuable when real data exists. It does not show that continuous touch transfers from simulation, and we do not compete in that regime. The 17-switch agents of Enhancing Tactile-based Reinforcement Learning for Robotic Control show that sparse binary inputs plus temporal prediction can produce very fast dexterity, though only in simulation. That is a warning that larger alphabets may add little where temporal context already compensates, and our vocabulary-size sweep tests exactly this.

Research Program

The simulator-side alphabet is defined per taxel from contact quantities the rigid-body engine exposes. Let $f_n$ be the normal force and $f_t$ the tangential force projected onto the taxel's sensing axes. A code $c = q_K(f_n, f_t)$ takes one of $K$ values: no contact, plus $L$ normal-force bins crossed with a shear-direction state of negative, none or positive. The force bins are deliberately coarse and are placed at quantiles of the simulated force distribution. Ordinal agreement ("harder than") is more plausible between simulator and sensor than absolute newtons. $K=2$ recovers binary abstraction, and $L=1$ with shear recovers the ternary scheme of the in-hand translation paper.

On the real side, an encoder $g_\phi(x)$ maps the raw reading $x$ of a taxel and its neighbors to a distribution over the same $K$ codes. For the DIGIT, $x$ is the difference-image patch. For ReSkin, it is the magnetometer vector over a short window, which lets the encoder model hysteresis. The encoder is trained on bridge data from a bench rig, where a probe on a force/torque sensor presses and drags across the skin at known normal force and shear direction. Labels come from passing the measured probe force through the same $q_K$. We budget about 30 minutes of rig time per sensor.

On held-out bridge data we estimate the confusion matrix

$$M_{ij} = P\big(\arg\max g_\phi(x) = j \mid q_K(f_n,f_t) = i\big),$$

where $i$ is the true code and $j$ the predicted code. During simulated reinforcement learning, each taxel's true code $c$ is replaced by a sample from row $M_{c,\cdot}$, with the matrix perturbed across episodes. The policy therefore trains against the real encoder's actual error pattern rather than generic noise. To recalibrate after drift, we refit $g_\phi$ and $M$ on a few minutes of new rig data while the policy stays frozen.

Baselines

The naive reference point is the proprioception-only policy trained identically in simulation. Its reported number is 33% real pivoting success (Sim2Real Manipulation on Unknown Objects). It tells us how much of any gain comes from touch at all.

The strongest existing method is hand-thresholded binary contact, with ternary shear on ReSkin. We reimplement it on our hardware with the same simulator, objects, reinforcement learning algorithm, seeds and training steps. Its thresholds are tuned on the same bench rig data our encoder uses, so the comparison isolates "learned predictor over a larger alphabet" rather than "more calibration data". Reported numbers are 80% success on real pivoting of 16 unseen objects for binary masks (Sim2Real Manipulation on Unknown Objects) and 93% average out-of-distribution success for binary-normal/ternary-shear translation (Learning In-Hand Translation). On our new force-sensitive tasks there is no reported result found on Lacuna, so we measure the threshold baseline ourselves. Beating it means one of two things. Either we beat it on force-sensitive tasks by a margin larger than the seed-to-seed spread of real rollouts, using at least 50 rollouts per condition, while staying within that spread on the original pivoting and translation tasks. Or we match it on all tasks before drift and clearly beat it after drift.

Three ablations isolate the key ingredients. The first is a learned encoder at $K=2$, which separates "learned instead of thresholded" from "larger alphabet". The second is the full $K$-code system with generic symmetric code noise in place of the measured $M$, which isolates confusion injection. The third is a VQ-VAE codebook learned unsupervised on real readings, with simulator codes mapped by nearest physical label. This is the closest analog to the speech-translation recipe and tests whether physics-defined codes are necessary.

A continuous-input policy trained with heavy domain randomization over sensor gain, noise and offset is also run. It is given the same tuning budget as our method's $K$ sweep.

Why not just…?

Why not just tune a better per-taxel threshold, or adapt it online? This is the most serious threat for $K=2$. If a carefully tuned per-taxel threshold matches our learned encoder at $K=2$, then the "learned" part adds nothing at binary resolution. The first ablation measures this directly. A threshold, however, cannot produce more than two ordered levels from a hysteretic, temporally smeared signal. The case for learning rests on $K>2$ and on drift. If the learned encoder does not beat per-taxel thresholds at any $K$, we will report that binary thresholding is sufficient and that our contribution reduces to the confusion-injection result.

Why not just train on continuous readings with domain randomization? The pivoting study already found that continuous RGB and difference representations degraded on real objects where binary masks did not (Sim2Real Manipulation on Unknown Objects). The translation work avoided continuous magnetic readings because fast simulators cannot model hysteresis (Learning In-Hand Translation). We still run this baseline on the force-sensitive tasks, where its richer input has the best chance of paying off.

Why not just learn the codebook end to end, as in discrete speech alignment? Discrete speech alignment worked because both modalities were real data and a bridge dataset anchored the codes (Discrete Cross-Modal Alignment). Our simulator produces contact forces, not sensor readings, so an unsupervised real-side codebook has no simulator counterpart to align to. The VQ-VAE ablation tests this. We expect it to underperform because its codes mix in lighting and gel artifacts rather than separating contact physics.

Why not just fine-tune on a few real demonstrations? That reintroduces the scarce paired data this work is designed to avoid, and the real data must be collected again whenever the sensor drifts. We include a 20-demonstration fine-tuning comparison on one task to show how many demonstrations the bench calibration is worth.

Proposed Experiments

We use the two real benchmarks above: DIGIT tool pivoting on 16 unseen objects over hard and soft surfaces, and Allegro/ReSkin in-hand translation on out-of-distribution objects, including the 20° hammer tilt. To these we add two force-sensitive variants. In the first, pivoting must be done without exceeding a crush limit on deformable objects, verified by an instrumented object. In the second, translation must be done with grip-force minimization, scored by success and peak force. The VTDexManip Isaac Gym tasks serve as a cheap pre-screen. There we replace the real sensor with a separately simulated hysteretic sensor model, sweep $K \in \{2, 3, 6, 9, 12\}$, and select candidates before spending hardware time.

The first decisive check is the vocabulary-size curve on real hardware. Success and peak force are plotted against $K$, and the hypothesis is falsified if the curve peaks at $K=2$ on the force-sensitive tasks. The second is drift. We deliberately degrade each sensor, by shifting magnet alignment on ReSkin and swapping in a worn gel on the DIGIT. We then compare the frozen threshold baseline, our system before recalibration, and our system after a five-minute rig refit, with the policy unchanged throughout. The third is the confusion-injection ablation at the best $K$. If measured confusion does not beat generic noise, the claimed mechanism is wrong even if the larger alphabet helps. Finally, we test whether simulated force bins agree with real ones. We report the encoder's held-out confusion matrix and the rank correlation between simulated and measured forces on rig-replicated contacts. A larger vocabulary transferring despite poor absolute force agreement would support the ordinal-bin design.