Parsing Touch When the Camera Is Blind

When a robot hand closes on an object or pushes a peg into a hole, its own fingers and wrist block the camera at the moment precision matters most. The usual fix is to add touch sensors and feed their readings into the policy, the learned controller that maps sensor readings to motor commands. Recent visuo-tactile policies do this, and they still fail in a characteristic way. They take a sequence of contact events that never occurs in a successful demonstration, such as pressing hard before the peg has slid onto the chamfer, and carry on regardless, because nothing in the system checks whether the sequence of touches makes sense. We propose to check it, the way a morphological analyzer checks whether a string of word pieces forms a real word. The idea is to mine an inventory of valid contact transitions from successful demonstrations only, parse the live tactile stream against it while the camera is blocked, and treat a failed parse as the trigger for recovery. On a failed parse, the robot rewinds to the last validly parsed contact state and resamples the next chunk of actions.

The analogy comes from computational morphology, and it carries a known risk. In Morphological Segmentation Inside-Out, word structure is parsed as a tree, and a validity rule requires every intermediate node to be a real word. That rule suppresses spurious analyses and beats flat segmenters by 6.6 F1 points (85.55 F1 on the CELEX-derived treebank). The same rule also makes the parser helpless when a valid root is missing from the dictionary. Work on lexical resource dependency gives a vocabulary for exactly this risk: a system bounded by the coverage of its symbolic inventory. So the proposal is the filter together with a measured account of how it fails when the inventory is incomplete. The test case is novel peg shapes and clearances, which produce contact sequences that are valid but were never seen in the demonstrations.

Core Claim

A small inventory of contact transitions mined from successful demonstrations, with smoothing toward coarser contact categories when exact transitions are unseen, can serve as an online validity filter over tactile and force readings. Wrapped around an existing visuo-tactile diffusion policy, it should raise success and lower peak contact force on occluded contact-rich tasks. A diffusion policy here means a generator that produces action sequences by iteratively denoising random noise. The filter should not hurt tasks where vision is clear. Unlike learned failure monitors, it needs no labeled failures, and unlike abort-only monitors, it produces a recovery point.

Three properties distinguish this from the closest existing work:

  • Unit of analysis. It works on contact sequences and validates intermediate states, not only predicted outcomes.
  • Training data. It is built from success-only data.
  • Response to failure. It returns where to rewind, not just whether to stop.

Research Program

Contact tokens. We first turn touch into discrete tokens. Tactile force fields, the per-taxel shear and normal force maps that ManiFeel found most useful for insertion, are vector-quantized per sensor pad into a codebook of a few dozen contact tokens. Wrist force/torque readings are quantized in the same way. Each codebook also gets a coarse level that clusters tokens into roughly six classes, such as free, light touch, sliding, and pressing.

Two-level grammar. The composition is hierarchical:

  • pad tokens combine into a hand-level contact state;
  • in bimanual tasks, the two hand states combine into a task-level state.

The analogue of the "every node is a word" rule is that each hand-level state, and each transition between consecutive states at each level, must appear in the inventory mined from successful demonstrations.

Scoring and the reject rule. Hard membership would inherit the brittleness the morphology literature warns about, so transitions are scored with backoff:

$$s_t = \log\big[\lambda\,\hat p(z_t \mid z_{t-1}) + (1-\lambda)\,\hat p(c(z_t) \mid c(z_{t-1}))\big],$$

The symbols are defined as follows:

  • $z_t$ is the hand-level contact state at time $t$;
  • $c(\cdot)$ maps a state to its coarse class;
  • $\hat p$ are smoothed transition frequencies from the demonstrations;
  • $\lambda \in [0,1]$ trades specificity against coverage.

A parse fails when $s_t < \tau$ for $k$ consecutive steps. The threshold $\tau$ is set on held-out successful demonstrations so that 2% of valid transitions are falsely rejected.

Recovery. On a failed parse, the controller does three things:

  1. retracts along the executed path to the most recent time at which all levels parsed validly;
  2. draws a new action chunk from the diffusion policy with fresh noise;
  3. rejects any resampled chunk whose first step reproduces the rejected direction of motion.

Scope of activation. The filter runs only while an occlusion estimate is high. That estimate is the fraction of the projected contact interface hidden from all cameras, computed from robot kinematics. This deliberately confines the filter to the regime where vision cannot verify progress.

Coverage study. The second half of the program studies lexical dependency directly, following the design of Targeted feature dropout. That paper varied dictionary coverage and showed that slot fillers collapse on unseen entities (80.66 F1 rising to about 94 only after dropout). We vary inventory coverage by mining from 10% to 100% of demonstrations and from subsets of object geometries. We then measure false rejections and task success on geometries never used for mining. If the filter is useful only at full coverage, that is a negative result we will report.

Baselines

Do-nothing reference. The naive reference is the underlying visuo-tactile diffusion policy run alone. It uses ManiFeel's force-field configuration in simulation and DECO's tactile-adapter policy (DECO) for bimanual data. Every other method wraps the identical trained checkpoint, so differences come only from monitoring and recovery. ManiFeel reports this policy family gaining 26 percentage points over vision-only on peg insertion and reaching 69% success on object search under occlusion. To beat this baseline, our filter must raise success on the occluded tasks by a margin larger than seed variance across at least three training seeds, without lowering success on clear-view tasks.

Force-threshold abort. The industrial heuristic stops when wrist force exceeds a tuned limit, then retracts and resamples with the same rewind logic as ours. Its limit is tuned on the same held-out success data under the same 2% false-trigger budget. This isolates whether contact-sequence structure adds anything over a scalar.

Strongest learned competitor. This is a ContactGuard-style monitor (ContactGuard): an action-conditioned latent world model with a logistic failure probe. That paper reports real-robot ROC AUC of 0.992 on cup and 0.946 on box manipulation using about 250 labeled attempts per task. We give it the same rewind-and-resample recovery and its full labeled budget, including failures that our method never sees, and the same compute for training. Beating it means matching its failure-detection AUC on our tasks and exceeding its end-task success once both drive recovery, while using no failure labels. Matching it with zero labels would itself be a useful outcome.

Force-aware policy reference. As a separate reference on real force data, we cite FAVLA (FAVLA): 80.8% average success, against 67.0% for ForceVLA and 42.8% for vision-only π0. This shows what a policy that ingests force directly achieves without any filter.

Ablations. Three ablations isolate the key ingredients:

  • a flat bigram filter over hand-level states with no task-level composition, mirroring the flat-versus-tree comparison in the morphology paper;
  • a hard membership filter with $\lambda = 1$ and no backoff;
  • the filter run at all times rather than only under occlusion.

Why not just…?

Why not just rely on the policy, since it already sees touch? DECO and FAVLA both inject touch or force into the policy and gain substantially from it. Yet FAVLA's authors report that their model still mistakes high misalignment forces for successful seating under some geometry variations. ManiFeel shows tactile inputs lowering performance by 5–6% when vision is adequate. Feeding touch in does not guarantee that the policy checks the order of contact events. The do-nothing comparison on the identical checkpoint settles this. If the wrapped policy is no better, the claim fails.

Why not just use a force limit? FAVLA's account of misalignment forces being read as success suggests that the magnitude of force is the wrong signal and its sequence is the right one. ContactGuard argues that post-contact thresholds fire too late. Still, a tuned limit plus retry is cheap and may capture most of the gain on single-arm insertion. The matched-false-trigger comparison will show whether it does. We expect structure to matter most on bimanual and reorientation tasks.

Why not just train a failure predictor like ContactGuard? This is the real threat. With enough labeled failures, a learned probe may detect bad chunks before contact, which our filter cannot do because it reacts to observed touch. Our position rests on two points. First, collecting labeled failures in occluded contact-rich tasks is costly and can be destructive. Second, ContactGuard is abort-only by design, while our parse supplies a rewind target. If the probe with recovery dominates at equal labels, the honest conclusion is that our filter is a label-free fallback, and the experiments are designed to make that visible.

Why not just a flat sequence model of contact? If a flat bigram filter performs as well as the hierarchy, we will adopt it. The hierarchy is expected to help only where per-hand states compose, as in DECO's bimanual tasks.

Proposed Experiments

Simulation. Simulation uses the ManiFeel task suite in TacSL (TacSL), with these tasks:

  • peg reorientation under occlusion;
  • object search under occlusion;
  • USB and power-plug insertion;
  • gear assembly under clear and dim lighting, as a no-harm control.

For each method we report success, peak contact force, jam count, and recovery success after rejection.

Coverage test. Lexical coverage is tested with the TLA tactile-action data (TLA). That paper reports over 85% success on unseen peg geometries and 1.0–2.0 mm clearances. We mine the inventory from a subset of shapes and measure the false-rejection rate on held-out shapes as a function of the backoff weight $\lambda$.

Hierarchy test. The hierarchy is evaluated offline on DECO-50. Because all 8,000 trajectories are successful, false rejection on held-out tasks is a clean measure of over-pruning. We compare tree and flat filters at equal rejection budgets on synthetically corrupted trajectories, where contact segments are swapped or reversed.

Real-robot check. Finally, a small zero-shot real-robot insertion check follows TacSL's sim-to-real protocol, which reports 82.7% zero-shot peg insertion.

Decisive checks. The proposal succeeds if three things hold together:

  1. the filter improves occluded success over the unwrapped policy and over the force limit;
  2. it leaves clear-view gear assembly unchanged;
  3. on unseen geometries, backoff keeps false rejections near the 2% budget, while the hard filter does not.

It fails if success gains vanish once the inventory is mined from fewer geometries than it is tested on. That failure would show that the lexical bottleneck transfers from morphology to touch intact.