Bootstrap Disagreement at Contact Transitions as a Predictor of Closed-Loop Tactile Performance

Touch sensors are now often evaluated the way word embeddings were a decade ago. A pretrained encoder is frozen, small "probe" models are trained on top of it to predict force, slip or pose from recorded data, and the encoder with the lowest offline error is assumed to be the best one for a robot. Natural language processing learned that this assumption can fail badly. Word-similarity scores showed little to no correlation with downstream task performance, and settings tuned for them actively hurt downstream tasks. Tactile sensing already shows hints of the same problem. A reconstruction-trained touch encoder can do fine offline and still fail when the robot actually moves. We propose a specific explanation and a cheap test for it. A robot controller does not suffer from a probe's average error. It suffers from how arbitrary the probe's prediction is at the few moments when contact changes: touch-down, the onset of slip, release. At those moments a small change in training data can flip the output. The robot then chatters or drops the object, even though average error barely moves.

The test borrows bagging (bootstrap aggregation), a classic way to measure and reduce a learner's sensitivity to its training data. We retrain the probe many times, each time on a version of the training set resampled with replacement, and measure how much the copies disagree. Our claim is that this disagreement, measured on frames near contact transitions, ranks tactile encoders by closed-loop manipulation success better than mean offline error does. We further claim that bagging the controller's tactile head helps most for the encoders the metric flags. That second claim turns a correlation into an intervention.

Core Claim

For a frozen tactile encoder $f$, we train $B$ small probe heads $g_1,\dots,g_B$. Each head is trained on a block-bootstrap resample, meaning whole contact episodes are drawn with replacement rather than individual frames, so temporal continuity is preserved. For a contact phase $p$, the instability of $f$ is

$$ U_p(f) = \frac{\mathbb{E}_{x \in \mathcal{D}_p}\, \mathrm{Var}_b\!\left[g_b(f(x))\right]}{\mathrm{Var}_{x \in \mathcal{D}_p}\!\left[y(x)\right]}, $$

where $\mathcal{D}_p$ is the set of held-out frames in phase $p$, $y(x)$ is the ground-truth target (for example, 3-axis force), and the denominator makes $U_p$ comparable across tasks. The phases (free space, onset, steady contact, slip, release) are labeled automatically from force ground truth. We predict that $U_{\text{transition}}(f)$, the instability pooled over onset, slip and release frames, has a higher Kendall rank correlation with closed-loop success than mean probe error does. We also predict that $U_{\text{transition}}$ adds predictive power after controlling for that error.

This is a measurement claim first. The proposal does not argue that bagging is the best way to make tactile policies robust. It argues that bootstrap disagreement at contact transitions is the missing intrinsic quantity that separates offline scores from deployment.

Why This Is Not Already Done

Three tactile representation papers supply the encoders and benchmarks, and all three evaluate on mean offline error. Sparsh pretrains touch encoders with masked reconstruction (MAE), self-distillation (DINO) and latent prediction (JEPA). It evaluates them on TacBench, a suite of offline probes plus the closed-loop Bead Maze task, and reports that latent models traverse "20-53% more distance" than baselines. It never measures how well probe scores rank closed-loop outcomes across models. AnyTouch and AnyTouch 2 train static-plus-dynamic representations, the latter with frame-difference video reconstruction and force regression on the 2.4M-sample ToucHD set. They show that encoders built for material recognition fail at insertion. However, they assume that lower force RMSE among dynamic encoders translates into better manipulation.

The two groups disagree about pixel reconstruction. Sparsh blames it for wasting capacity on glare and gel wear. AnyTouch 2 calls it necessary for millisecond-scale slip. Our metric offers a concrete way to settle this. Reconstruction-based encoders may reach low mean error while being unstable precisely at transitions, because lighting artifacts give a probe many equally good but conflicting ways to fit.

Several lines of work motivate the gap but do not supply a tactile metric. In language, Faruqui and Tsvetkov and Schnabel et al. documented the gap. Chiu showed that tuning for similarity degrades named entity recognition. Adi et al. showed that surface probes fail to forecast semantic task success. The closest constructive precedent is QVEC, which found an intrinsic score that tracks downstream success by correlating embedding dimensions with linguistic features. We play the same role for touch, but our score measures stability rather than alignment. Activation with Intrinsic-Extrinsic Consensus uses the words "intrinsic" and "extrinsic" for activation gating inside a classifier, which is unrelated to evaluation. "Extrinsic contact" in Neural Contact Fields is physical contact between a grasped object and the environment, also unrelated.

Bagging work supplies the method and its constraints. Cooper et al. and RashomonGB frame variance across near-equal models as arbitrariness, or "predictive multiplicity". That is exactly the quantity we measure, but they study it only in tabular fairness settings. Ensemble-SINDy bags over trajectory segments to recover equations from noisy physical time series, which is the precedent for resampling whole segments instead of single rows.

Liang, Zhu and Zhang constrain the design in a decisive way. They found that stable learners such as nearest neighbors and Naive Bayes show almost no bagging effect. Bagged linear probes would therefore show little disagreement for reasons unrelated to the encoder. We use small multilayer perceptron probes, which they identify as both unstable and accurate enough to benefit from bagging. We also check that the ranking survives a change in probe width.

Research Program

Encoder pool. We first build a pool of roughly 24 frozen tactile encoders that differ in ways that matter: the Sparsh MAE, DINO and JEPA variants with and without temporal frame stacking, AnyTouch and AnyTouch 2, an end-to-end trained ResNet-18, and intermediate pretraining checkpoints. Checkpoints matter because the NLP lesson is that selection within a family is where intrinsic metrics mislead most.

Intrinsic scores. Each encoder receives the standard intrinsic scores: TacBench force, slip, pose and grasp-stability probe errors, and Sparsh Bench tangential force RMSE. Each also receives the new phase-stratified bootstrap disagreement, with $B=16$ heads trained under an identical budget. The resampling unit is the contact episode. A second variant resamples contact-phase blocks, following the open question of whether naive resampling destroys the temporal derivatives needed for slip.

Extrinsic scores. We then measure closed-loop success for every encoder with the encoder frozen and a fixed small policy head trained by imitation:

Intervention. Finally, for each encoder we compare a single policy head against a bagged head that averages actions from $B$ heads trained on episode bootstraps. If instability is causal, the bagging gain should grow with $U_{\text{transition}}$ and be near zero for encoders the metric scores as stable.

Baselines

The naive reference point for prediction is to rank encoders by mean force error on TacBench, which is what current papers effectively do. Sparsh reports force error "below 0.1N" for its DINO variant. However, no reported result was found on Lacuna for the rank correlation between any TacBench probe score and Bead Maze or insertion success. Establishing that correlation is itself our first deliverable. It is run on the same encoders, the same probe architecture and the same training data as our metric.

The strongest existing intrinsic predictor is the full TacBench battery combined into one score. We will use the better of two combinations. One is the principal-component combination of intrinsic metrics proposed in the evaluation literature. The other is a regression of success on all probe errors, cross-validated over encoders. Beating it means a higher held-out Kendall correlation with a bootstrap confidence interval over encoders that excludes the battery's value. In addition, $U_{\text{transition}}$ must keep a significant partial correlation once mean error is controlled for.

The key ablations isolate each ingredient. Disagreement pooled over all frames, rather than transition frames, tests whether timing matters. Seed-only ensembles, which use the same data with different initializations, test whether data resampling matters beyond optimization noise. Frame-level resampling versus episode-level resampling tests whether preserving temporal structure matters.

For the intervention, the strongest existing method for tactile or force-conditioned control in the file is FACT. Its time-aware force injection uses a temporal convolutional network over a 2-second history and raised average success from "41%" to "66%" across five contact-rich tasks. For a plug-in tactile adapter, DECO reports "82.50%" average success on its bimanual suite. We include a FACT-style 2-second history encoder as a policy-head variant under the same demonstrations and rollout counts. Bagging is judged as a complement to it, not a replacement.

Why not just…?

Why not just run closed-loop rollouts for every candidate and skip intrinsic metrics? That is the ground truth, and we do it here for 24 encoders. It does not scale to checkpoint selection or hyperparameter sweeps, which is why the field relies on proxies. The proxy is worth having only if it ranks well. That is exactly what the held-out correlation test decides. If mean error already ranks as well as our metric, we will report that the gap does not exist among dynamic tactile encoders, which is itself a finding.

Why not just use seed ensembles or dropout uncertainty instead of bootstrap resampling? This is the most serious threat. Seed variance is cheaper, and in deep networks it often dominates resampling variance. The seed-only ablation settles it directly. If seed disagreement predicts success as well as episode-bootstrap disagreement, the contribution narrows to "transition-phase disagreement predicts deployment", and we would recommend the cheaper estimator. The argument for data resampling comes from Cooper et al. and RashomonGB. In their framing, arbitrariness arises from which data the model saw, and rare transition frames are exactly where a resample changes that most.

Why not just smooth the policy's actions over time, or feed it more history, instead of bagging? Temporal smoothing and history encoders such as FACT's network address chattering directly. They may make the intervention arm redundant. We therefore run bagging both alone and on top of the 2-second-history head. The measurement claim does not depend on bagging being the best fix. If smoothing captures the gain, $U_{\text{transition}}$ should still predict which encoders need it.

Why not just add a slip-detection probe to TacBench? A slip probe's mean accuracy is another average-error score. It is exactly what Sparsh already reports and what our baseline battery includes. Our hypothesis is that variance at transitions, not accuracy on them, is what the battery misses. The partial-correlation test separates the two.

Proposed Experiments

Correlation study. For each of the 24 encoders, we compute every intrinsic score on TacBench and Sparsh Bench, along with $U_p$ for each phase. We then run 30 rollouts per encoder on Bead Maze and on each insertion task, about 2,200 rollouts in total, comparable to FACT's roughly 2,500. We report Kendall correlation with success, using leave-encoder-family-out cross-validation so that one family's checkpoints cannot inflate the result.

Pixel-versus-latent contradiction. We test whether Sparsh-MAE and AnyTouch 2's frame-difference objective show low mean error but high $U_{\text{transition}}$. A reversal would undercut Sparsh's account. Agreement would explain the contradiction between the two papers without either being wrong about their own benchmark.

Intervention. We regress the per-encoder bagging gain on $U_{\text{transition}}$, measured on a separate rollout batch. We expect a positive slope, with negligible gain for stable encoders, in line with Liang et al.'s finding that stable learners gain little from bagging.

Decisive check. The proposal fails if $U_{\text{transition}}$ does not beat the best TacBench combination on held-out encoders, or if its predictive power disappears once mean error is controlled for. It also fails, in its strong form, if seed-only disagreement matches it. All three outcomes are informative, and the rollout data will be released alongside offline scores as the first public benchmark of how well tactile intrinsic evaluation predicts closed-loop outcomes.