Robots that learn manipulation from demonstrations face two separate problems. First, a policy trained on one arm tends to break on another. Second, a policy that relies on cameras tends to break once the hand blocks the camera's view of the object it is holding. Work on object-centric task representation addresses the first problem. These methods describe the scene and the actions relative to the gripper or the objects, not in a robot's joint angles, so the learned behavior does not depend on the arm's kinematics. Work on early feature fusion addresses the second. These methods mix vision and touch inside the network before any decision is made, so each signal can correct the other. So far the two lines have not met. The policies that transfer across arms see only through cameras. The policies that use touch well feed it in as a fixed-size vector tied to one particular sensor mounted on one particular gripper.
We propose a simple bridge. Each tactile reading becomes a 3D point, placed where the contact actually happens. It joins the same gripper-frame point cloud that the visual features already live in. The point carries a contact flag and a shear vector, the sideways deformation of the sensor skin, which signals slip, rotated into that same frame. One encoder then processes visual and tactile points together. The policy predicts the grasped object's motion relative to its target. Because touch is expressed as geometry and not as a sensor-specific channel layout, the claim is that one visuotactile policy can survive changes of arm, finger shape, sensor placement, and even taxel count (the number and layout of sensing cells). Vision-plus-touch token policies cannot do this without retraining.
Core Claim
The claim is narrow and testable. In distribution, on contact-rich insertion and turning tasks, lifting touch into the gripper-frame scene cloud should match a well-tuned "tactile token" policy. It should not lose to it. Under changes to the gripper or tactile hardware, however, it should keep most of its success while the token policy collapses. It should also beat a vision-only gripper-frame policy whenever the in-hand object is occluded or slips.
If the lifted representation gives no advantage under hardware change, the idea has failed. Gaining accuracy in distribution would not rescue it.
Relation to Prior Work
The method builds directly on Adapt3R. Adapt3R lifts CLIP image features into a metric point cloud, expresses that cloud in the end-effector frame, and feeds it to standard imitation policies. This gives cross-arm and cross-camera transfer at 44.1 Hz. We reuse its lifting and downsampling, so the cross-arm part of our transfer result is largely inherited from it. What we add is a contact modality that lives in the same coordinates. Adapt3R itself has no touch, which is exactly what fails under occlusion.
Three visuotactile systems motivate the touch side, and each shows a different limitation:
- TacThru-UMI fuses wrist RGB, see-through tactile images, and marker shear into one diffusion-policy transformer. It reports 85.5% average success, against 55.4% for vision only and 66.3% for touch only. This is strong evidence that early fusion helps. Its tactile input, however, is tied to one handheld gripper's layout.
- TouchGuide fuses late. A contact model's gradients steer a visual policy's denoising steps, raising Diffusion Policy from 16.3% to 36.2% on five contact-rich tasks. Its actions stay in base-frame coordinates, and the visual features never learn from touch.
- VTDexManip shows that binarizing contact, keeping only "where" and not "how much", regularizes sim-to-real transfer. It reports a roughly 40% total gain over vision only.
TacThru's argument for continuous shear pulls in the opposite direction from VTDexManip's binarization. We keep both channels on each tactile point and ablate them separately.
Two further works constrain the design:
- Cross-Tactile Sensor Representation Learning maps different optical tactile sensors into one shared feature space. It aligns what a sensor reading looks like, not where on the hand the reading occurs. We treat it as a complementary per-point encoder, not as a competitor.
- Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies shows that relative-pose actions transfer from Franka to Kuka at 0.1 mm tolerance. It depends on CAD-model visual tracking that fails under hand occlusion, the failure our contact points are meant to cover.
Research Program
The representation works as follows. At each control step, multi-view RGB-D is lifted and transformed into the gripper frame as in Adapt3R. Each active taxel, or each region of an optical tactile image, is mapped to a 3D point using the sensor's known pose on the finger. Its features are a contact flag, a pressure value, and a shear vector rotated into the gripper frame. A learned modality embedding is added, then a sensor-agnostic appearance code obtained from a CTSRL-style encoder when the sensor is optical.
Visual and tactile points then enter one point-transformer encoder. Its attention is restricted to each point's local neighborhood. A small set of learned queries reads out a summary for the action head. The local restriction is how we address the latency tension the file raises: dense attention across all tokens is costly, while local attention lets a contact point modulate the visual points near it without paying full quadratic cost. Our target is at least 30 Hz, against Adapt3R's 44.1 Hz without touch.
Because tactile points are just more points, a sensor with a different taxel count or placement changes only the point set. The network's input shape stays the same.
For the action, a diffusion policy head predicts the motion of the gripper relative to a target region. The target is segmented from the lifted cloud by the language-conditioned CLIP features. The predicted motion is executed through inverse kinematics on whichever arm is present.
The analysis has three parts. First, we measure how often the policy attends from visual points to nearby contact points in occluded phases versus free-space phases. This is a direct test of whether fusion happens where it should. Second, we probe whether the encoder's latent state predicts in-hand object displacement after slip. Third, we compare binary-only, shear-only, and full tactile points, to settle the VTDexManip versus TacThru tension on tasks with and without slip.
Baselines
The naive reference point is a vision-only gripper-frame policy: Adapt3R with the same diffusion head, the same demonstrations, and no touch. If our method cannot beat this on occluded contact phases, touch is adding nothing. On the insertion tasks we also report the pure potential-field controller from Ramesh et al. as a floor. That controller falls to 3% under sensor noise at 0.1 mm tolerance in their paper.
The strongest existing competitor is a TacThru-style early token fusion, run inside Adapt3R's gripper frame. In this baseline, the tactile signals pass through the same per-sensor encoder and are appended as global tokens with fixed sensor identity. They fuse with the visual tokens in a shared transformer. This is the right opponent because it contains the same information and the same fusion depth as our method; only the geometric placement differs. On TacThru's own hardware, the reported multimodal average is 85.5%. For our task suite there is no reported result found on Lacuna for this exact configuration, so we will run it ourselves. Both methods get identical demonstrations, identical encoders, the same number of training steps, and an equal hyperparameter search budget of the same number of configurations.
We also compare against TouchGuide-style late steering applied to the vision-only Adapt3R policy. TouchGuide reports a gain from 16.3% to 36.2% for Diffusion Policy on its suite; we will implement its contact model on our tasks.
The key ablation removes geometric placement from our own method. Tactile points keep their features, but their coordinates are replaced by a fixed per-sensor anchor. This isolates whether location in the frame matters.
Beating the token baseline means two things. In distribution, the gap must be no worse than a few points. Under the hardware-shift conditions below, our method must show a statistically clear advantage across at least two of the three shift types, with enough seeds and trials for confidence intervals on the difference to exclude zero.
Why not just…?
Why not just append tactile tokens to an Adapt3R policy? This is the real threat. In distribution, it may perform just as well, because the network can learn the fixed sensor geometry implicitly. Our proposal stands or falls on the hardware-shift experiments. With a different finger length, a relocated pad, or a different taxel grid, a token policy receives inputs whose meaning has silently changed, or whose dimension no longer fits. The lifted representation receives points in new but correct places. If the token baseline holds up under these shifts as well, we would report that geometric lifting is unnecessary.
Why not just steer a vision policy with touch at inference time, as TouchGuide does? Late steering is modular and needs no retraining of the visual policy, which is attractive. But the visual features never learn from contact. During occlusion, the visual policy's proposals are therefore drawn from a distribution that does not know where the object sits in the hand. The occluded-insertion condition in our experiments compares the two directly. TouchGuide's own Lock Opening result, 30% with 20 demonstrations, suggests that late steering leaves a large share of failures unrecovered.
Why not just use a sensor-agnostic tactile encoder like CTSRL and keep tokens? That approach normalizes how different sensors' images look, but not where on the hand a contact occurred, and not the direction of shear in task coordinates. The latter is what determines the corrective motion. We include CTSRL-style per-point encoding inside our method. We also run the token baseline with the same encoder, so that any difference between them reflects geometry alone.
Why not just add a wrist force-torque sensor? A wrist force-torque sensor gives one net force and torque. That cannot tell whether a peg is touching the hole's left or right edge, which a spatial contact map can. We include a vision plus force-torque variant on the insertion tasks as a cheaper-hardware check.
Proposed Experiments
Simulation. We extend the IsaacGym tight-insertion setup from Ramesh et al. (tolerances 2 mm down to 0.1 mm) with simulated fingertip taxel pads. Pads report contact flags from contact forces and shear from tangential forces, in the binary-contact style VTDexManip uses in Isaac Gym. We add VTDexManip's BottleCap Turning and Lever Sliding tasks. Policies are trained by imitation from scripted and residual-RL demonstrations collected on a Franka gripper with one pad layout.
We evaluate three kinds of shift. The first is arm swap, Franka to Kuka, as in Ramesh et al. The second is gripper and sensor shift: longer fingers, pads displaced 5–10 mm, and pad resolution changed from 4×4 to 8×8 taxels. The third is induced in-hand slip with camera occlusion of the grasped part. Metrics are success rate, final insertion error, and control frequency.
Real robot. We run cap insertion as in TacThru, three-prong charger insertion as in Ramesh et al. (who report 10/10 zero-shot on Kuka), and lock opening from TouchGuide. Optical fingertip sensors are mounted on a Franka gripper, with 20 trials per condition. The decisive transfer test retrains nothing. We swap one finger's sensor for a different optical model at a different mounting position, and separately move to a second arm.
Decisive checks. Three outcomes decide the proposal. First, under the gripper and sensor shift, our method should retain most of its in-distribution success while the token baseline and the anchor ablation lose substantially. Second, under occlusion with slip, our method should beat vision-only Adapt3R and the TouchGuide variant. Third, control should stay at or above 30 Hz.
If check one fails, the contribution reduces to "touch helps gripper-frame policies," which TacThru already suggests, and we would say so. The binary versus shear ablation on the slip conditions is reported regardless of outcome. It will show whether VTDexManip's finding that "where" matters more than "how much" survives once slip matters.
References
- Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learning
- Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation
- TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance
- VTDexManip: A Dataset and Benchmark for Visual-tactile Pretraining and Dexterous Manipulation with Reinforcement Learning
- Cross-Tactile Sensor Representation Learning
- Integrating Model-Based Control and RL for Sim2Real Transfer of Tight Insertion Policies