Retrieving Simulated Recoveries by Contact State for Sim-to-Real Manipulation

Robots trained in simulation often fail in the real world at the moment of contact. A peg jams, a grasped object starts to slip, or a soft object deforms differently than the physics engine predicted. The camera image at that moment may look nearly the same in simulation and reality, but the forces do not. We propose indexing a library of simulated demonstration fragments by contact state, meaning short, nameable physical situations such as "jammed against an edge," "sliding under shear," "compressing a compliant surface," or "seated." A small learned module maps the real robot's high-rate force and touch signals into the same embedding space as text descriptions of those states. When the real robot enters a contact state, it looks up simulated fragments that begin in the same state and conditions its next actions on how simulation resolved that situation.

The bet is that contact states are abstract in the way "justice" is abstract for a vision-language model. They have no localized visual referent, yet they may be more stable across the simulation-to-reality gap than either pixels or raw force traces. The precise force magnitudes will differ between sim and real, but "the tool is jammed" should mean the same thing in both. If that holds, retrieval keyed on contact concepts should transfer better than retrieval keyed on visual scene similarity, which is what current retrieval-augmented robot policies use. The work tests this directly and includes the ablation that could show the concept layer is unnecessary.

Core Claim

Retrieval-augmented robot policies currently choose what to retrieve by looking at the scene. STRAP segments offline demonstrations wherever end-effector velocity dips, then matches sub-trajectories using features from DINOv2 (a self-supervised image encoder) and subsequence dynamic time warping. Retrieval-Augmented Embodied Agents (RAEA) retrieves whole policies from a memory bank using global ImageBind embeddings plus proprioceptive state. RALAD retrieves real driving feature maps to patch simulated perception, and never acts. These systems motivate our proposal: STRAP lifts few-shot real success on DROID-Kitchen from 10% (behavior cloning) to 61%, and RAEA's real-robot success falls from 54% to 36% without proprioception. Retrieval clearly helps, and non-visual signals clearly matter. None of them, however, keys retrieval on what is physically happening at the contact.

Our claim is that the appropriate key for sim-to-real retrieval in contact-rich tasks is a projected contact-state concept, and that segmenting the memory at contact-state transitions rather than velocity dips is what makes retrieved fragments reusable. Concretely, we train a projector $f$ that maps a window of wrench and tactile signals $x_{t-w:t}$ to a vector aligned with frozen text embeddings $e(c)$ of contact-state descriptions $c$. Simulation supplies the supervision for free, because the physics engine reports contact normals, slip velocity, penetration, and stuck/sliding friction modes at every step. We then test whether the concept key survives physics shift better than visual or raw-force keys.

Research Program

The first component is a contact-state vocabulary and an automatic labeler. We define roughly a dozen states, including free motion, first touch, stable grasp, incipient slip, sliding, edge jam, wedged, compliant compression, and seated. We derive them from simulator ground truth in ManiSkill2 (ManiSkill2), whose soft-body tasks use a coupled material-point physics model, and in ManiSkill-ViTac 2025 (ManiSkill-ViTac 2025), which simulates GelSight Mini tactile sensors with a finite-element contact model. Each state receives several paraphrased text descriptions. The text serves two purposes: new states can later be specified by description alone, and the text embedding space gives nearby states such as "slip" and "shear" nearby keys. Iosif et al. observe that abstract concepts are organized by attributional similarity across semantic neighborhoods, and graded similarity of this kind is what we want the keys to carry.

The second component is the projector. We keep the full sensing rate: 1 kHz wrench and the tactile displacement field, encoded by a small temporal convolution. We do not subsample, because Video-LLaVA keeps only 8 frames and explicitly loses rapid state shifts. There is also direct evidence that high-rate signals matter: the JIGSAWS analysis finds that training hours and global ratings correlate poorly with skill, while derivative kinematics such as jerk carry task-specific signal. The projector is trained contrastively against frozen text embeddings, following the practice of training only the bridging layer between frozen encoders, as in Multimodal Few-Shot Learning with Frozen Language Models and Dong et al.. We apply heavy randomization of friction, stiffness, and sensor gain in simulation. A small calibration set of about a hundred hand-labeled real contact windows is used only to fit a temperature and an abstention threshold, never the projector weights.

The third component is the memory and the policy. We segment simulated demonstrations wherever the ground-truth contact state changes and store each fragment under the projected embedding of its starting state, together with its wrench-domain action sequence. We predict wrench rather than position because TacDiffusion showed that force-domain diffusion policies transfer zero-shot to tight-clearance insertion, with 95.7% success. Our base policy is a TacDiffusion-style wrench diffusion policy that cross-attends to the top-$k$ retrieved fragments, similar to how RAEA conditions on retrieved trajectories.

Retrieval is triggered only when the projected state changes with confidence above the calibrated threshold. This follows RaDIO, which fires retrieval from an internal detector instead of at every step. The gate responds to the warning in U-NIAH that capable generators degrade when handed irrelevant retrieved context, and it also addresses the latency that RAEA flags as a bottleneck.

Baselines

The naive reference is the same wrench diffusion policy trained in simulation with domain randomization and no retrieval, deployed zero-shot. It shows how much of the gap the base policy already closes. For calibration, ManiSkill2 reports that a sim-trained PickCube policy reaches 91% in simulation and 60% zero-shot on a real robot.

The strongest existing competitor is STRAP-style retrieval, meaning velocity-dip segmentation with DINOv2 and subsequence dynamic time warping. It reports 61% real success on DROID-Kitchen, against 31% for a multi-task policy and 10% for few-shot behavior cloning. It is the right competitor because it is the best-documented sub-trajectory retrieval method with real-robot results. We also run RAEA-style retrieval (ImageBind plus proprioception), which reports 69% real success with multimodal inputs.

To keep the comparison fair, all retrieval variants share our memory of simulated fragments, the same policy network, the same top-$k$, and the same gating budget. Where a baseline normally retrieves continuously, we also run it with our gate, so that gating is not a hidden advantage. Every method receives the same hyperparameter search budget and the same hundred real calibration windows. For STRAP and RAEA the calibration windows go toward threshold tuning.

Three ablations isolate the key ingredient. The first retrieves on raw wrench/tactile windows using dynamic time warping, with no concept projection. The second replaces the text-aligned projector with a plain discrete classifier over the same states, which asks whether language alignment matters or only the discretization does. The third keeps concept keys but segments by velocity dips, which isolates the segmentation choice.

Beating the strongest baseline means a statistically reliable improvement in real success over STRAP-style retrieval, on at least 50 trials per task, on the physics-divergent tasks described below. It also means no loss on rigid insertion, where TacDiffusion-style policies are already strong.

Why not just…?

Why not just retrieve on raw force traces? This is the most serious threat. If dynamic time warping over wrench windows retrieves the right fragments, the concept layer adds nothing. Our reason to expect otherwise is that raw magnitudes shift with real friction and stiffness, while state identity should not. Still, this is an empirical question, and the first ablation settles it. If raw-trace retrieval matches concept retrieval under physics shift, we would drop the projector and report contact-transition segmentation as the contribution.

Why not just randomize physics and train end-to-end? TacDiffusion's 95.7% shows that end-to-end wrench policies can transfer well on rigid insertion, so we do not expect retrieval to help there. The case for retrieval rests on soft and frictional contact, where approximated physics diverges (von Bechtolsheim et al.; Gajbhiye) and a single policy has to average over mismatched dynamics. The naive baseline, split by rigid versus soft task, measures exactly this.

Why not just fine-tune on real data? We include a variant that fine-tunes the base policy on the same small real budget spent as demonstrations. RALAD reports that frozen-encoder retrieval avoids forgetting and cuts training time roughly tenfold (34.84 s to 3.81 s per epoch) compared with retraining in perception. We will check whether the same holds for control, and we will report it honestly if fine-tuning wins at equal real-data cost.

Why not just put the whole history in a long context? This fails on latency for kilohertz control. U-NIAH also finds that retrieval beats pure long-context processing 82.58% of the time for 7B/8B-scale models, and that recall collapses as distractors grow. We therefore do not test a long-context variant for control.

Proposed Experiments

The decisive first experiment is sim-to-sim and fully controllable. We build the memory under one physics setting in ManiSkill2 soft-body tasks and ManiSkill-ViTac insertion. We then execute under progressively shifted friction, stiffness, and sensor gain, and measure two things. The first is contact-state retrieval accuracy: whether the retrieved fragment starts in the true state. The second is task success. The prediction is that concept-keyed accuracy degrades more slowly with shift than both DINOv2 keys and raw-trace keys. If it does not, the core claim is false before any hardware is involved. ManiSkill-ViTac has no reported result found on Lacuna, so all numbers on it will be our own runs.

The second experiment is real-robot transfer on three task types. Rigid insertion uses TacDiffusion's geometries, tested zero-shot at clearances of 0.02 to 0.05 mm. Compliant pressing uses foam and silicone. Grasp-and-lift includes objects prone to slipping. Each is run against all baselines and ablations with the fairness controls described above.

The third experiment probes concept grounding directly. We hold out one contact state, for example "wedged," from projector training. We add it to the vocabulary by text description only and test whether the projector can recognize it and retrieve from it. A success here would be the only result that justifies aligning the projector with language rather than training a plain classifier.