Subtracting the Hand's Own Motion: Event-Based Slip Sensing Inside Occluded Dexterous Grasps

When a multi-fingered robot hand closes around an object, its fingers block every camera's view of the contact. The robot loses sight of the object at exactly the moment it matters most. Event cameras are a natural tool for this blind spot. An event camera is a sensor whose pixels independently report brightness changes with microsecond timing, and it keeps working in deep shadow and glare. Prior work found that a camera fixed to a gripper sees almost nothing while a grasp is stable, because the held object does not move relative to the sensor, and sees a burst of events the instant the object slips. That "silence means secure" signal works for a two-jaw gripper that stops moving once closed.

It breaks for a dexterous hand. A 16 to 22 joint hand keeps moving its fingers while it squeezes, adjusts and reorients, so its own fingers flood the event stream. We propose to recover the silence signal by subtracting it back out. The robot knows its joint angles and its hand geometry. From these it can predict which events its own fingers will produce, cancel them, and treat whatever events remain as evidence that the object is moving against the hand. The idea borrows from the "efference copy" in biology, where an animal discounts sensations it caused itself. The residual stream then drives a fast corrective reflex, which closes the open-loop terminal phase of current dexterous grasping policies.

Core Claim

Our claim is that, for a palm- or wrist-mounted event camera on a high-DoF hand, a proprioception-driven forward model that predicts and removes self-generated events produces a slip signal that is both faster and more specific than three alternatives: raw events, events fed to a network alongside joint states, or frame-based tactile and RGB sensing. The specific failure we target is false alarms during deliberate finger motion. We expect this advantage to grow with the number of moving joints, and to be largest in the dark, shadowed cavity under the palm, where frame cameras underexpose.

The contrast with prior work is concrete.

  • Evetac. Evetac puts an event camera inside a tactile elastomer on a parallel gripper and stabilizes grip force at 500 Hz. It never faces self-generated finger motion, and it senses only the contact patch.
  • Event slip simulator. The combined physics and event simulator relies on the stationary-object property for slip classification on a Franka gripper. It does not close the loop, and its accuracy fell from 80% on synthetic data to 63% on real data.
  • DECO and ADEPT. DECO and ADEPT handle occlusion in high-DoF hands with frame-rate tactile sensing. They motivate the problem but do not touch event streams.
  • ContactGuard. ContactGuard shapes our design more than any other paper. It found that feeding joint states into a visual world model hurt prediction, because the network used proprioception as a shortcut. We therefore use proprioception to explain away sensory input rather than as an extra input. This is a direct, testable answer to that negative result. It also sits alongside DECO's opposite finding that proprioception is essential as a global conditioning signal.

Research Program

Sensor. The setup is a small event camera (the DVXplorer Mini class used in Evetac) mounted in or just behind the palm. It looks into the space the fingers close around.

Forward model. At each instant, we render the hand mesh into the camera from the current and commanded joint angles. This gives the predicted log-intensity change $\hat{L}(x,t)$ at each pixel $x$ caused purely by finger motion. The residual event density in an object region $\Omega$ over a short window $[t-\tau, t]$ is then

$$ s(t) = \frac{1}{|\Omega|}\sum_{x\in\Omega} \Big| E(x,t) - \mathbb{1}\big[|\hat{L}(x,t)| \ge C\big]\, E(x,t) \Big|, $$

where $E(x,t)$ is the observed event count at pixel $x$ in the window and $C$ is the camera's contrast threshold. This slip score $s(t)$ counts events not explained by the hand.

We study three versions of the forward model in order of complexity:

  • a binary kinematic mask that discards all pixels covered by moving fingers;
  • the analytic predicted-event cancellation above, which keeps object pixels that border fingers;
  • a small learned residual network that corrects the rendered prediction for shading and specular effects. It is trained in simulation on hand-only motion with no object, so it cannot learn about slip.

Simulation. Training data and controlled tests come from Isaac Sim with the v2e event simulator, following the slip simulator pipeline. We extend that pipeline to ShadowHand and Allegro and use objects and grasps from DexGraspNet. Lighting is randomized from 5 lux to strong point specular.

Reflex. The slip score feeds a reflex layer on top of an unchanged base grasping policy, DemoGrasp, whose terminal phase currently runs open loop. When $s(t)$ crosses a threshold, the reflex applies a bounded finger-closure and wrist correction. That correction is learned as a residual on DemoGrasp's own trajectory-edit action space, so control stays low-dimensional even though the hand is not.

What the project contributes. The deliverables are the cancellation model, the reflex, and a measurement of how much the cancellation matters as the number of moving joints and the darkness of the scene increase.

Baselines

Do nothing. The naive reference is DemoGrasp executed open loop through the terminal phase, as published. Its reported real-world grasp success is 86.5% overall and 71.1% on objects thinner than 1.5 cm (DemoGrasp). We re-run it on our own hand, objects and lighting rather than reusing those numbers, since the hardware will differ.

Raw-event reflex. The second baseline is the parallel-gripper rule transplanted unchanged: trigger on total event density in the object region with no cancellation, as in the slip simulator. We expect it to fire constantly during finger motion. Measuring how often it does is the core motivation for the project.

Strongest learned competitor. The most serious competitor is an event-plus-proprioception network. It receives the same event stream and the same joint states as our method and is trained end-to-end to predict slip, using the same data, the same compute budget and the same hyperparameter search. Proprioception is injected by Adaptive Layer Normalization, the route DECO found necessary. This baseline isolates our key ingredient. It has all the same information and could in principle learn the cancellation itself. Our method beats it only if explicit subtraction gives a better false-alarm rate at matched detection latency, especially on hand motions and objects held out from training.

Frame-based tactile. We use the Inspire hand's tactile pads, the sensing that drives DECO's 82.5% average success on DECO-50 versus 72.25% vision-only (DECO). They feed the identical reflex. ADEPT's tactile student, which reports 8/10 on asymmetric FMB peg insertion (ADEPT), is a further reference for what tactile sensing achieves, though on a different task.

Event tactile on the gripper. For slip detection alone, the reference is Evetac's 93% and 88% slip-timing success on known and novel objects (Evetac), measured on a parallel gripper.

Frame camera in the same spot. A palm-mounted global-shutter RGB camera at 60 to 120 Hz receives the same kinematic mask and uses frame differencing. This separates the benefit of event sensing from the benefit of camera placement.

Ablations. Ablations remove the learned residual and swap the binary mask for analytic cancellation.

Why not just…?

Why not just give the network events and joint states and let it learn the cancellation? This is the most serious threat, and the strongest competitor above tests it directly. Our reason for doubt is ContactGuard's finding that networks use proprioception as a shortcut instead of reading visual dynamics. In our setting, a network could learn "fingers closing implies events implies no slip" and miss slips that happen during closure. If the learned model matches ours on held-out finger motions, we will report explicit cancellation as unnecessary. In that case we would reposition the forward model as an auxiliary training target rather than a pipeline stage.

Why not just use fingertip tactile sensors, which are not occluded at all? Tactile pads and gel sensors sense only where contact already exists, and the optical ones run at 30 to 60 Hz. Evetac shows that micro-slip vibrations reach hundreds of Hertz. A palm event camera also sees the object before and between contacts, including rotation of the object about a fingertip that a pad cannot localize. The tactile-pad baseline runs on identical hardware and the identical reflex. If pads match us under normal light, our contribution narrows to the low-light and high-speed regimes. We will report that outcome rather than hide it.

Why not just use a high-speed RGB camera with a hand mask? This is the frame camera in the same spot, included as a baseline. The case against it rests on the palm cavity being dark and specular. Event sensors have tracked state down to 10 lux (Sun et al.) and reconstructed scenes at 7 lux where RGB failed (Dynamic EventNeRF), and they need a small fraction of the bandwidth (Evetac reports 1.7% of RGB tactile data). Whether this matters at the distances inside a grasp is an empirical question, settled by the lighting sweep below.

Why not just abort with a world-model monitor? ContactGuard reaches ROC AUC 0.992 on its cup task. However, it acts before contact and only aborts, so it cannot catch slip that begins during the squeeze. We report its pre-contact AUC as a reference but do not treat it as a substitute for recovery.

Proposed Experiments

Simulation sweep. The first experiment runs in simulation and asks whether cancellation is needed and when. Hands span 1 effective DoF (a parallel-gripper control), 16 DoF (Allegro) and 22 DoF (ShadowHand). Slips are induced by randomized friction and external pulls during grasp, lift and in-hand reorientation. We measure:

  • slip-detection ROC AUC;
  • false alarms per second during deliberate finger motion with no slip;
  • detection latency from slip onset.

The decisive check has two parts. Raw events and our method should perform similarly on the parallel gripper. The gap between them should widen with joint count. If no widening appears, the core claim is wrong.

Real-world slip detection. We mount the palm camera on an Allegro or Inspire hand and repeat the slip-detection measurements under three conditions: room light, 10 lux, and a single harsh spot light producing specular highlights on metal objects. Ground-truth slip comes from external motion capture on the object. This also tests whether cancellation narrows the 80% to 63% sim-to-real drop reported by the slip simulator. The rendered hand prediction should transfer better than learned appearance does.

Closed loop. We run the DemoGrasp base policy with each reflex variant on a held-out object set that includes thin items and low-friction surfaces. We measure grasp-and-lift success, drop rate under scripted perturbations (string pulls during lift), and the number of unnecessary corrections.

Success criteria. Success means three things against the matched learned competitor and the tactile-pad reflex on the same objects and lighting:

  • a lower drop rate than the open-loop DemoGrasp baseline;
  • no loss in clean-grasp success from spurious corrections;
  • an advantage that is largest at 10 lux.

A null result in the dark condition would mean that HDR is not what makes event sensing useful here. That finding would be worth reporting as well.