Robots that learn manipulation by copying human demonstrations increasingly carry both cameras and touch sensors. The hope is that touch takes over once the gripper meets the object, because touch registers contact, force and slip that the camera cannot see. There is a quieter failure hiding in this setup. In recorded demonstrations, the moment of contact is almost perfectly predictable from vision alone: the gripper is a certain number of pixels from the peg, a shadow closes, the fingers occlude the object. A policy trained to copy expert actions can learn "switch to squeezing when the picture looks like this" instead of "switch to squeezing when the fingers feel the object." Both rules fit the training data equally well. They diverge only at deployment, when an object sits a few millimeters off, is softer than usual, or the lighting changes. In those cases the visual cue and the physical contact come apart, and a vision-timed policy squeezes air or crushes the egg.
We propose to treat this as a specific, measurable case of causal confusion: a copying policy mistakes a correlate of the true cause of an action for the cause itself. We also propose a fix that needs no environment labels and no online trial and error. The key observation is that every demonstration dataset already contains natural "disagreement moments," frames where a vision-only contact predictor and the actual touch signal disagree because of small variations in pose, compliance, or occlusion. Grouping frames by the sign of this disagreement produces exactly the environment split that invariance methods need. A policy that works equally well across these groups cannot be relying on the visual proxy for contact.
Core Claim
Visuo-tactile imitation policies built on standard fusion architectures time their contact-phase actions from visual proxies rather than from touch. The effect is small on in-distribution tests and large under shifts that move physical contact relative to its visual appearance. It can be removed offline by training for equal action-prediction loss across pseudo-environments defined by vision–touch disagreement. The new element is the source of the environments. Invariant imitation methods such as Invariant Causal Imitation Learning for Generalizable Policies (ICIL) split latent state into causal and nuisance parts using adversarial losses over given environment labels, such as hospital IDs or simulator domains. Robot teleoperation logs have no such labels. We derive them from the cross-modal structure of the data itself.
This connects the three directions in unequal roles. Causal confusion is the phenomenon under study. Invariance across environments is the mechanism. Visuo-tactile fusion is the regime that makes the problem acute, and it supplies the second modality that lets us build environments without supervision.
Why this gap is real
Two findings motivate the proposal. DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter keeps touch in a separate cross-attention adapter because concatenated visual tokens "drown out" sparse tactile cues. That is an architectural symptom of vision absorbing the contact signal. ManiFeel: Benchmarking and Understanding Visuotactile Manipulation Policy Learning benchmarks 13 visuo-tactile tasks and reports that adding tactile images to vision-sufficient gear assembly lowers success by 5–6%. It attributes this to overfitting on sensor artifacts, which is spurious reliance running in the opposite direction. Neither paper asks which modality the policy's contact decisions actually depend on, or tests under shifts designed to separate them.
The causal-confusion literature supplies the lens but not the modality. Causal Confusion in Imitation Learning showed that extra observations can hurt closed-loop performance. It resolved this with graph masks disambiguated by expert queries or environment rollouts, which is costly on a physical robot handling fragile objects. Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning (OREO) works offline by masking whole visual objects found by a discrete autoencoder. RoCoDA: Counterfactual Data Augmentation for Data-Efficient Robot Learning from Demonstrations swaps causally irrelevant scene parts under rigid-body transforms. All three operate on images or low-dimensional state. None has a way to perturb a tactile field consistently with contact physics, and none targets the timing relation between two modalities. On the fusion side, Visuo-Tactile Transformers for Manipulation adds auxiliary contact and alignment embeddings, and 3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing projects tactile arrays into the camera point cloud. These methods improve how touch is represented. They do not constrain what the action depends on.
Research Program
The work has three parts: a diagnostic, a method, and an assumption check that decides whether the method applies to a given dataset.
The diagnostic: a contact-lag stress test. In the ManiFeel simulation suite (IsaacGym with TacSL tactile simulation), we can move the true contact event without much change to its visual appearance. We do this by displacing the target object 1–5 mm along the approach axis, by changing its compliance so contact registers at a different fingertip depth, or by adding lighting shifts to the occlusion cue. The primary metric is action-onset misalignment: the time between true tactile contact onset and the policy's switch into its contact-phase behavior. The switch is detected as the change point in commanded grip force or approach velocity. Task success and a damage proxy, peak contact force above a fragility threshold, are secondary metrics. A policy that times its actions from touch should show misalignment near zero regardless of the displacement. A vision-timed policy should show misalignment that grows roughly linearly with displacement divided by approach speed. That slope is the diagnostic signature.
The method: disagreement-stratified invariance. We first train a vision-only contact predictor $\hat c_v(o^{vis}_t)\in[0,1]$ that estimates, from the image at time $t$, whether the fingers are in contact. To avoid leakage, it is cross-fitted over demonstration folds, so each frame is scored by a model that never saw it. Each frame is then assigned to one of three groups $g\in\{\text{agree},\ \text{vision-early},\ \text{vision-late}\}$ by comparing $\hat c_v$ with the thresholded tactile contact signal $c^{tac}_t$. The policy $\pi$, which can be any fusion architecture, is trained with
$$ \mathcal{L}(\pi)=\frac{1}{3}\sum_{g}R_g(\pi)\;+\;\lambda\,\mathrm{Var}_g\big[R_g(\pi)\big], $$
where $R_g$ is the standard imitation loss (the diffusion denoising loss for diffusion policies) averaged over frames in group $g$, and $\lambda$ sets how strongly unequal group losses are penalized. The disagreement groups are rare but decisive. In "vision-early" frames the picture says contact but the fingers feel nothing, so the expert has not yet switched. A policy that uses the visual proxy will predict the switch too early and incur high loss only there. The variance term keeps that loss from being averaged away by the many easy "agree" frames.
We will also test a representation-level version. It adds ICIL's adversarial group classifier on the policy's pre-action embedding, with our pseudo-groups replacing ICIL's environment labels. This tests whether ICIL's machinery transfers once the labeling problem is solved. The file flags estimator instability in high dimensions as a known limitation of ICIL-style disentanglement, so we treat this variant as secondary.
The assumption check: does the expert act on contact? The method is only valid if the demonstrator's contact-phase actions track physical contact rather than their own visual cue. A human teleoperator without haptic feedback may well be vision-timed, and then the "confused" policy is faithfully copying a vision-timed expert. We test this directly. For each demonstration, we measure whether the expert's action change points align better with tactile onset or with $\hat c_v$ onset. In simulation we control the answer by using a privileged-state scripted expert, which is touch-timed by construction. On DECO-50 (DECO-50 dataset), 50 hours of bimanual teleoperation with 17 tactile pads per hand, we report the measured alignment. If DECO-50 experts turn out to be vision-timed, that is itself a useful finding: tactile adapters trained on such data are learning something other than reaction to touch, and our method should not be applied there.
A secondary concern is that post-contact force readings partly reflect the robot's own previous grip command. This is the "copycat" structure described in Fighting Copycat Agents in Behavioral Cloning from Observation Histories. We will log whether the method shifts reliance from vision onto this self-caused tactile component, so that we do not trade one shortcut for another.
Baselines
The naive reference is a vision-only diffusion policy, together with the same policy using simple concatenation of visual, tactile and proprioceptive features. For concatenation, Visuo-Tactile Transformers for Manipulation reports plateaus of 40–60% success on simulated door opening, against about 80% for its own model. Concatenation is the setting where we expect the visual contact shortcut to be strongest, so it anchors the size of the effect.
The strongest existing fusion method is DECO's decoupled design, with vision–action self-attention and tactile cross-attention through a plugin adapter. On DECO-50 it reports an 82.5% average success rate with the tactile adapter, against 72.25% for vision and proprioception alone. It is the strongest competitor because it is explicitly built to stop vision from swamping touch. If architectural decoupling already fixes contact timing, our loss adds nothing. Within ManiFeel, the relevant reported numbers are that tactile force fields add 26 percentage points over vision-only on peg insertion, and that tactile images cost 5–6% on nominal gear assembly. We found no reported result on Lacuna for any method under contact-lag shifts, so all shifted-condition numbers will be produced by us.
We will run every method, DECO-style included, on the same demonstrations, the same visual and tactile encoders, the same number of gradient steps, and the same hyperparameter search budget (a fixed number of trials per method, selected on held-out unshifted validation demonstrations only, so that no method tunes to the stress test). Our loss is applied on top of both the concatenation and the DECO-style architecture, so that architecture and objective effects can be separated.
Two ablations isolate the key ingredient. The first reweights the disagreement frames to equal total mass with no variance penalty, in the style of group-balanced resampling. It tests whether the benefit comes from simply seeing rare frames more often. The second assigns random pseudo-groups of the same sizes, which tests whether the benefit depends on the disagreement structure at all. Beating the baselines means reducing the slope of action-onset misalignment against displacement by at least half relative to DECO-style fusion. It also means improving shifted-condition success without losing more than 2 points of nominal success, and without the regression on vision-sufficient gear assembly that ManiFeel documents.
Why not just…?
Why not just use modality dropout, randomly blanking vision during training so the policy must use touch? This is the most serious threat. It is roughly OREO's masking idea applied at the modality level rather than the object level. Its weakness is that blanking vision removes the global context the policy still needs after contact, such as where to move and what the next subgoal is. It also does nothing about a policy that, when vision is present at test time, still prefers the visual timing cue. The file offers only indirect evidence: in Causal Confusion in Imitation Learning, plain dropout did not fix confounded behavioral cloning. That evidence comes from a different setting, so we do not rely on it. Modality dropout at several rates is a full baseline under the stress test. If it matches our misalignment slope at equal nominal success, we will report that and drop the heavier method.
Why not just hard-code "see-then-feel," switching control to touch the instant contact registers? The hierarchical division of labor described in ManiFeel and VTT suggests this gating would fix timing by construction on single-contact insertion tasks, and we expect it to do well there. It breaks where contact is intermittent or multi-point, as in bimanual sorting or in-hand regrasping, and where vision must remain in the loop after contact, as in the occluded object-search task. We include a gated policy as a baseline and report results separately for single-contact and multi-contact tasks. If gating wins on both, the contribution shrinks to the diagnostic.
Why not just collect more varied demonstrations, or augment them counterfactually? More variation in object pose would itself decorrelate visual proxies from contact, and RoCoDA shows how strong counterfactual augmentation is for visual distractors (99.3% versus 22.3% on Libero-Object). But RoCoDA's rigid-body swaps cannot produce the matching tactile field for a displaced or softer object, and extra physical demonstrations are costly. Our method uses the variation already present in the data. To test directly whether data volume alone closes the gap, we will train the concatenation baseline on double the simulated demonstrations with widened pose randomization.
Proposed Experiments and Decisive Checks
The first experiment establishes that the problem exists. On ManiFeel peg insertion, gear assembly and object search, we measure the misalignment-versus-displacement slope for vision-only, concatenation and DECO-style policies. The proposal stands or falls here. If fusion policies already show near-zero slope, visual contact shortcuts are not a real problem at this scale and we stop.
The second experiment checks the assumption. We measure expert action alignment with tactile versus visual onset in scripted simulated data and in DECO-50.
The third experiment is the main comparison: every baseline and ablation under the stress test, at three displacement magnitudes and two compliance levels, with 100 rollouts per condition and bootstrapped confidence intervals.
The fourth experiment is an offline check on DECO-50, restricted to frames in the disagreement groups. It compares action-prediction error on disagreement frames against agreement frames across methods. This is the closest we can come to real-robot evidence without hardware. It is interpretable only if the second experiment shows that DECO-50 experts are touch-timed.
A clean positive result would look like this: DECO-style fusion still shows a clear misalignment slope, the variance-penalized loss removes most of it while random groups and plain reweighting do not, and modality dropout and hard gating each fail on at least one task family. Any other pattern tells us which simpler fix is enough.
References
- Invariant Causal Imitation Learning for Generalizable Policies
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- ManiFeel: Benchmarking and Understanding Visuotactile Manipulation Policy Learning
- Causal Confusion in Imitation Learning
- Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning
- RoCoDA: Counterfactual Data Augmentation for Data-Efficient Robot Learning from Demonstrations
- Visuo-Tactile Transformers for Manipulation
- 3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing
- Fighting Copycat Agents in Behavioral Cloning from Observation Histories