Robots that grasp soft objects or insert connectors need both sight and touch. A common recipe trains them in two stages. First, the robot copies human demonstrations (imitation learning). Then it improves by trial and error against a reward (reinforcement learning, RL). A separate line of work fuses camera and touch signals with cross-modal attention, a layer in which features from one sensor act as "questions" that pick out the relevant parts of another sensor's features. Joining the two recipes looks natural, but it carries a specific risk. The early RL updates are noisy, and they are driven by a value estimator that is still poorly trained. Vision is the easier signal to optimize. So these updates may quietly overwrite the attention weights that read touch, and the robot slides back into a vision-only policy at exactly the moments, contact and self-occlusion, when touch matters most.
We propose to measure this "tactile drift" directly and to prevent it with a structural choice. After imitation pre-training, we freeze the vision–action backbone. Online RL may update only two things: a small low-rank adapter inside the cross-attention block through which vision reads touch, and a residual action head. The claim is that this restriction makes fine-tuning more sample-efficient and more robust to perturbation than fine-tuning the whole fused network or residual RL on concatenated features. The reason given is that it preserves, and sharpens, the policy's use of touch. If that holds, which parameters RL may touch becomes a design decision for multisensory policies, rather than "all of them" by default.
Core Claim and How It Differs From Prior Work
Three nearby papers each hold two of the three ingredients. Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs trains a vision-queries-touch attention module with Soft Actor-Critic from scratch. That paper motivates our proposal in two ways. Its attention policy beats early and late fusion under perturbation, and its authors also report that the extra attention parameters make the policy sensitive to environment randomization. We read that sensitivity as a sign that RL gradients alone fit high-capacity fusion poorly. Touch begins where vision ends (VITAL) does visuotactile behavioral cloning followed by residual RL with DrQ-v2, reaching about 90% success in unseen environments with 32 demonstrations and 45 minutes of online RL. Its encoders, however, are merged without cross-attention, so it gives our closest recipe but says nothing about attention stability. DECO supplies the architectural method: a frozen vision–action backbone with a plug-in tactile cross-attention adapter tuned by low-rank adaptation (LoRA, which updates a small low-rank correction to a weight matrix instead of the full matrix), touching 7.97M of 91M parameters for a 20% absolute gain on contact-rich tasks. DECO is imitation-only. Its ablations also constrain our design: swapping which pathway carries touch versus body state caused collapse. We therefore keep DECO's routing fixed and do not search over it.
Our proposal differs from all three in one respect. It treats the cross-modal adapter as the interface through which RL is allowed to act, and it tests whether that interface prevents the fused policy from losing touch during fine-tuning. Prior work does not measure that loss.
Research Program
The policy follows DECO's decoupling, scaled down to the simulated setting. Image tokens and action tokens share self-attention. Proprioception modulates layers through adaptive layer normalization. Touch enters only through a cross-attention adapter in which visual tokens query tactile tokens. Pre-training is behavioral cloning on demonstrations. To fine-tune, we use residual DrQ-v2 as in VITAL, with one change. The trainable set is restricted to LoRA weights on the adapter's query, key and value projections, plus a small residual action head. The backbone stays frozen.
The critic, the network that estimates future reward, gets special treatment. Distilling Motion Planner Augmented Policies shows that pairing a pre-trained actor with a randomly initialized critic can destroy the actor. We therefore initialize the critic from Monte Carlo returns on the demonstrations for every method we compare. This keeps critic anchoring from confounding the comparison.
The analysis tool is a tactile reliance index, tracked throughout fine-tuning. It is computed separately for states in contact and states without contact, using the simulator's contact flags:
$$R_t = \mathbb{E}_{s \in \mathcal{S}_c}\big[\lVert \pi_t(s) - \pi_t(s_{\tilde{\tau}}) \rVert\big],$$
Here $\pi_t$ is the policy after $t$ RL updates, $\mathcal{S}_c$ is a fixed held-out set of contact states, and $s_{\tilde{\tau}}$ is the same state with its tactile reading replaced by one drawn from a different episode. We also report the success-rate drop under the same tactile shuffling, and the entropy of attention over tactile tokens. Tactile drift is defined as $R_t$ falling below its value at the end of behavioral cloning while visual reliance rises. The central hypothesis has two parts. Full fine-tuning drifts. Adapter-only fine-tuning holds $R_t$ steady or raises it in contact states while leaving it near zero in free space.
Baselines
The do-nothing reference is the pre-trained behavioral cloning policy with no RL. It tells us how much any online phase adds.
The strongest from-scratch competitor is DRQ-CMA from Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs. On its PyBullet+TACTO setup it reports nearly 100% grasp success in the standard condition, against about 5% for vision-only. In the "Random Move" zigzag condition it reports 60%, against 35–45% for early and late fusion. We rerun it with its own architecture and the same total environment-step budget as our method. Because pre-training uses no environment steps beyond the demonstrations, we also report the demonstration count. To count as beating DRQ-CMA, our method must reach its standard-condition success in clearly fewer steps, and it must exceed 60% under Random Move.
The strongest two-stage competitor is a VITAL-style pipeline: the same demonstrations, behavioral cloning on concatenated visual and tactile features, and residual DrQ-v2. The file has no reported VITAL result on PyBullet+TACTO, so we reproduce it ourselves. Its only reported figure is roughly 90% success on real contact-rich tasks. It gets the same critic initialization, step budget and hyperparameter search budget as our method.
The obvious combination is the same cross-attention policy with every parameter unfrozen during RL. A variant of it adds modality dropout, randomly blanking a sensor during training, so that we test the standard remedy for modality dominance.
Two ablations isolate the key ingredient. One freezes everything, including the adapter, and trains only the residual head; this asks whether adapter plasticity matters at all. The other swaps the trained set to LoRA on the vision self-attention instead of the tactile adapter, with the same parameter count. Adapter-only RL beats the obvious combination only if it wins on Random Move success and steps-to-80%, and only if its tactile reliance index does not decline where the full fine-tune's does.
Why not just…?
Why not just do residual RL on concatenated features, as VITAL does? This is the most serious threat. VITAL already reaches about 90% on real insertion without attention. Our evidence against concatenation comes from settings without imitation pre-training. There, Lee et al. find early fusion falls to 35–45% under Random Move, and DECO reports interference when touch and vision are lumped together. It is entirely possible that a behavioral-cloning start removes concatenation's disadvantage. The head-to-head on Random Move and on held-out object sizes settles this. If concatenation matches us there, we will report that attention adds nothing once a policy is warm-started.
Why not just fine-tune everything with modality dropout? Dropout forces each pathway to be usable alone. It does not stop RL gradients from re-weighting attention toward vision when both sensors are present, which is the drift we care about. The dropout arm, together with the tactile reliance index, tests this directly. If dropout keeps $R_t$ stable and matches our success, the restriction to adapter weights is unnecessary.
Why not just freeze the whole pre-trained network and learn only a residual action head? That is the safest option against forgetting. However, it cannot improve how touch is read, for example when contact dynamics under Random Move differ from those in the demonstrations. The freeze-all ablation measures how much adapter plasticity buys. A null result would shrink our claim to "freeze and add a residual," and we would say so.
Why not just add Posterior Behavioral Cloning noise? Posterior Behavioral Cloning shows that ordinary behavioral cloning gives overconfident policies that cover valid actions poorly. Its fix injects noise in proportion to demonstrator uncertainty, and it reaches 75% success on Robomimic roughly twice as fast as standard behavioral cloning. That method addresses action coverage, while ours addresses which parameters RL may change. The two are complementary, so we run it as an add-on to both our method and the full fine-tune rather than as a rival.
Proposed Experiments
The primary testbed is the PyBullet+TACTO setup from Lee et al.: a 6-DoF arm with a Robotiq 2F-85 gripper and six deformable objects, in the standard condition and the Random Move condition. Demonstrations come from a scripted grasp controller, in sets of 10, 30 and 100 per object, so we can see how the effect depends on demonstration count. Metrics are success rate against environment steps, steps to 80% success, final Random Move success, and the tactile reliance and attention-entropy curves. We train on four object sizes and test on two held-out sizes. All results use five seeds with confidence intervals.
The first decisive check is whether tactile drift exists at all. If full fine-tuning keeps $R_t$ flat and matches adapter-only success, the motivating mechanism is wrong. The second is whether adapter-only RL beats both the VITAL-style pipeline and the freeze-all ablation under Random Move. The third is whether the vision-LoRA swap underperforms the tactile-adapter version at equal parameter count; if so, the benefit comes from where plasticity is placed, not from how much there is.
If simulation results hold, a small real-robot replication follows the VITAL protocol, with 32 demonstrations and a 45-minute online budget per task on a visuotactile insertion task. DECO-50 cannot support online RL because it is offline humanoid data. We use it only to check offline whether DECO's adapter attention shows the same contact-phase selectivity that our reliance index rewards.
References
- Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs
- Touch begins where vision ends: Generalizable policies for contact-rich manipulation
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning