Touch Where It Counts: Gaussian-Process Variance as a Spatial Anchor Against Visual Dominance in Visuo-Tactile Shape Reconstruction

When a robot reconstructs an object from camera images plus a handful of fingertip touches, the cameras supply millions of pixels and the touches supply a few dozen small patches. A learned model that fuses the two through attention, where each queried 3D point decides how much to "look at" visual features versus tactile features, tends to drift toward the plentiful, easy-to-fit visual stream. This is a spatial version of a well-known failure called modality dominance, in which a multimodal network learns mostly from one input and ignores the rest. The result is a model that, at the exact spot the finger touched a shiny or transparent surface, trusts a confidently wrong camera depth over a direct physical measurement. We propose to fix this with a signal the field already knows how to compute: the posterior variance of a Gaussian-process implicit surface fitted to the touches alone. That variance gives a calibrated map of where touch actually constrains the geometry.

The proposal in two sentences: fit a tactile-only Gaussian process to the contacts, and use its spatially varying confidence to bias and floor the attention each 3D query point pays to tactile tokens, so attention to touch is forced to be high near informative contacts and near zero elsewhere. This makes the rebalancing local rather than global. That matters because the standard fixes for modality dominance push the neglected modality's weight up everywhere, which in sparse-touch reconstruction injects noise into the large regions no finger ever reached.

Core Claim

Modality dominance in dense geometric regression is a spatially local pathology, and global rebalancers mis-target it. Gradient Blending (What Makes Training Multi-Modal Classification Networks Hard?) reweights per-modality losses by how fast each overfits. FULLER (FULLER) slows the dominant encoder's gradient. RollingQ (RollingQ) rotates queries toward the neglected modality using one rebalance anchor. Each corrects the imbalance with a scalar or a single direction shared across all inputs. In visuo-tactile reconstruction the correct tactile share varies by orders of magnitude across space. It should be close to one on a touched patch of a mirror-like surface and close to zero on the unobserved back of the object.

We claim that replacing RollingQ's single anchor with a spatially varying target derived from Gaussian-process posterior variance does two things that neither global rebalancing nor analytic Bayesian fusion achieves. First, near contact, the reconstruction error stays at or below that of the tactile-only Gaussian process regardless of how many camera views are added. Second, far from contact, accuracy matches or beats additive fusion. If both hold, the approach also resolves the open disagreement between VTacO (Visual-Tactile Sensing for In-Hand Object Reconstruction), which found that adding tactile features beats richer fusion because sparse touch pollutes non-contact regions, and ViTacFormer (ViTacFormer), which found cross-attention necessary. On our account each is right in a different region of space.

Research Program

The base model is a coordinate-query implicit decoder. Here "implicit" means the shape is represented as a function returning, for any 3D point, its signed distance to the surface. Each query point $x$ cross-attends over a set of visual tokens, taken from multi-view image or point-cloud features, and a set of tactile tokens, one per contact patch, encoded as in VTacO from DIGIT-style tactile images. The decoder outputs a mean signed distance $\mu(x)$ and a predictive variance $s^2(x)$, trained with a Gaussian negative log-likelihood so that it reports its own uncertainty.

In parallel, and with no learning, we fit a Gaussian-process implicit surface to the contact points alone. This follows the tactile Gaussian-process fits used in Touch-GS (Touch-GS) and in task-informed grasping (Task-Informed Grasping of Partially Observed Objects). The fit gives a tactile posterior variance $\sigma_t^2(x)$ at every query point. We define tactile confidence as

$$c(x) = 1 - \frac{\sigma_t^2(x)}{\sigma_{0}^2},$$

where $\sigma_0^2$ is the Gaussian-process prior variance, so $c(x)$ runs from 0 (touch says nothing here) to near 1 (touch pins the surface down).

The confidence enters the attention in two ways. The first is an additive bias of $\log(c(x)+\epsilon)$ on the logits of tactile tokens, which suppresses tactile attention where touch is uninformative. This recovers the stability of additive fusion in non-contact regions. The second is a training-time hinge penalty that activates whenever the total attention mass on tactile tokens, $a_t(x)$, falls below a floor proportional to confidence:

$$\mathcal{L}_{\text{anchor}} = \mathbb{E}_x\big[\max(0,\ \gamma\, c(x) - a_t(x))\big],$$

with $\gamma \in (0,1]$ the floor strength. This blocks the collapse RollingQ documents, in which attention keeps flowing to the dominant stream even when that stream is replaced by noise. The floor is only a lower bound. The model is still free to give touch even more weight, or to combine touch with vision in whatever way lowers error. That freedom is what separates the method from hard-coded precision weighting.

We will also build the diagnostic this setting lacks: a spatial attention-imbalance curve. It plots mean tactile attention mass $a_t$ against binned tactile confidence $c$. A healthy fused model should show a monotone curve. A collapsed model shows a flat curve near zero. This is a spatial analogue of RollingQ's Attention Imbalance Rate, and it lets us measure dominance in reconstruction directly instead of inferring it from final error.

Baselines

The naive reference point is the tactile-only Gaussian-process implicit surface itself, alongside a vision-only version of our decoder. The tactile-only surface sets the error a fused model must not exceed near contact. The vision-only decoder sets the error far from contact. A fused model that loses to either in its own region is showing the multimodal performance drop that Wang et al. report on Kinetics, where joint late fusion reached 71.7% against 72.6% for RGB alone (What Makes Training Multi-Modal Classification Networks Hard?).

The strongest existing learned method on our main benchmark is VTacO (Visual-Tactile Sensing for In-Hand Object Reconstruction). It fuses a 3D U-Net over point clouds with ResNet-18 tactile features by addition and decodes a winding-number field. It reports an Intersection over Union of 0.860 on rigid objects in VT-Sim, against 0.622 for 3DVT. We will retrain VTacO with the authors' setup on the identical VT-Sim splits. Our decoder will use the same visual and tactile encoders, so the only difference is fusion. Both methods get the same hyperparameter search budget. Beating VTacO means a higher rigid-object IoU than our reproduced VTacO number. It also means a statistically significant drop in near-contact Chamfer distance, the average nearest-point distance between reconstructed and true surfaces, measured within a fixed radius of each contact, across three seeds.

The strongest non-learned competitor is analytic Bayesian fusion in the style of Touch-GS. It combines tactile Gaussian-process predictions with metric-aligned monocular depth by inverse-variance weighting. We give it the same inputs, including a visual uncertainty estimate from the vision-only decoder's variance head.

The rebalancing baselines use our plain cross-attention decoder, with no confidence bias and no anchor, under one of four existing fixes: Gradient Blending with per-modality auxiliary heads, FULLER-style intra-modality gradient gating, modality dropout as in Wolf et al., or RollingQ with its global anchor. Each receives the same tuning budget as our method.

The ablations that isolate the key ingredient fall into two groups. The first group tests each half of the mechanism: bias only, anchor only, and both. The second group tests whether Gaussian-process variance is the right signal. One ablation replaces $c(x)$ with a variance map shuffled across objects. Another replaces it with a confidence computed purely from distance to the nearest contact, the kind of proximity heuristic Snap-it, Tap-it, Splat-it uses (Snap-it, Tap-it, Splat-it).

Why not just…?

Why not just fuse analytically with inverse-variance weights, as Touch-GS does? If both modalities reported calibrated variances, this would be optimal and our learned attention would be redundant. That is a real threat. Our bet is that visual uncertainty is badly calibrated exactly where it matters: on specular and transparent surfaces, monocular and multi-view depth tends to be confidently wrong, which is the failure Touch-GS and Snap-it set out to address. Learned attention, floored by tactile confidence, can override vision where vision's self-reported variance cannot be trusted. The decisive test is the glossy and transparent object subset. If analytic fusion matches us there in near-contact error, the learned component is not justified and we will report that.

Why not just add tactile features to visual features, as VTacO does? Addition is cheap, and VTacO found it beat concatenation for sparse touch. Addition, however, gives the same tactile influence to every query point inside the feature volume it is added to. It cannot down-weight a misleading visual feature at a contact. Our view-scaling experiment, described below, measures whether additive fusion's near-contact error rises as views increase. That rise is the signature of dominance that addition cannot correct.

Why not just use modality dropout, Gradient Blending, or RollingQ? These are the natural reviewer suggestions, and they are run as baselines. Wang et al. already observe that generic regularizers fail to equalize modalities that learn at different rates. Our specific prediction is that the global methods will raise tactile attention in unobserved regions as well as at contacts. The spatial imbalance curve will show this directly as elevated tactile mass at low $c$, together with worse far-from-contact error than the vision-only decoder.

Why not just use distance to the nearest contact instead of a Gaussian process? This is the most serious threat to the probabilistic component, and the distance-based ablation tests it head on. Gaussian-process variance differs from distance in two ways: it shrinks faster where touches cluster, and it extends along the kernel's smoothness scale rather than isotropically. So the two signals should differ most when touches are dense on one face and absent elsewhere. If the distance version matches across all touch-density regimes, we will conclude that the simpler heuristic suffices and that the probabilistic machinery is not earning its cost.

Proposed Experiments

The primary benchmark is VT-Sim with YCB and AKB-48 rigid objects, as used by VTacO, with reported IoU and Chamfer distance computed separately within and beyond a contact radius. Secondary evaluation uses Glossy Synthetic to stress specular failure. There we compare against the published Snap-it, Tap-it, Splat-it figure of 0.0026 Chamfer distance at 5 views, with the caveat that Snap-it optimizes each scene separately whereas we decode in a single feedforward pass. That comparison is therefore indicative, not head-to-head. Real-world checks use YCB objects with a parallel or multi-finger hand carrying DIGIT sensors, following the protocol of the proactive-exploration study (Proactive tactile exploration).

The decisive experiment is a grid over visual view count, from 1 to 20, and touch count, also from 1 to 20. For each cell we record near-contact error relative to the tactile-only Gaussian process. Dominance appears as near-contact error that increases with view count for plain cross-attention, additive fusion, and the global rebalancers. Our method should keep that ratio at or below one in every cell.

Beyond the grid, we report the spatial imbalance curve for every method. We also report calibration of the decoder's predicted variance, using negative log-likelihood and the fraction of true surface points that fall within predicted intervals, because a reconstruction that claims to be probabilistic must be scored as one. Finally, as a downstream check, we feed the variance into the uncertainty-aware grasp objective of task-informed grasping and compare grasp success on held-out YCB objects. That paper reports above 90% success with vision-only Gaussian processes, which gives a reference point.

The two outcomes that would falsify the proposal are analytic inverse-variance fusion matching us on glossy objects, and the distance-based confidence matching the Gaussian-process confidence across touch-density regimes.