Measuring Finger Slip with Event Cameras to Remove Contact Skate in Hand-Object Motion

When a computer reconstructs or generates a hand handling an object, the fingers often glide across the surface they are supposed to be gripping. Character animation has the same problem with feet, where it is called "foot skating." Animation fixes it with a simple rule: predict when a foot touches the ground and penalize its velocity to zero during that time. That rule fails for hands. Dexterous manipulation is full of slip that is meant to happen: an object pivots between the fingertips, a finger slides along an edge, a grasp is loosened so the object can rotate. A zero-velocity rule cannot tell a skating artifact from a deliberate slide, so it erases the very motions that make manipulation interesting.

We propose to measure slip directly instead of guessing it. An event camera is a sensor whose pixels independently report brightness changes with microsecond timing, so it produces no output for static scenes and has no motion blur. Around each candidate fingertip contact, we compare the apparent motion of the event stream on the finger side of the boundary with the motion on the object side. Matching motion means the contact is sticking. Different motion means the finger is sliding, and the difference gives the slip velocity. That measured relative velocity then replaces the zero-velocity target in hand-object reconstruction and serves as training labels for hand-object motion generators. In two sentences: event cameras can measure how fast a finger slides relative to the object it touches, finely and without blur. Using that measurement as the contact target, instead of forcing contact velocity to zero, should remove finger skating while preserving intentional sliding.

Core Claim

The contact constraint that matters in manipulation is the relative tangential velocity at the contact patch, not the velocity of the finger. Locomotion hides this distinction because the ground is static. Our claim is that a confidence-weighted, event-derived estimate of this relative velocity, used as the target of a contact loss, reduces unintended contact slip during sticking phases more than binary contact labels with zero-velocity losses do. It should also track the velocity of intended slip during sliding phases, which the binary approach cannot represent at all.

Concretely, for contact point $k$ at time $t$, the method compares $v^{rel}_{k,t}$, the tangential velocity of the hand surface relative to the object surface as implied by the current hand and object pose estimates, against $\hat{s}_{k,t}$, the relative tangential velocity measured from events. The comparison is weighted by $w_{k,t}$, a confidence that grows with the number of events supporting the measurement:

$$ \mathcal{L}_{slip} = \sum_{k,t} w_{k,t}\, \lVert v^{rel}_{k,t} - \hat{s}_{k,t} \rVert^2 . $$

The classical foot-skate loss from Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control is the special case where $\hat{s}=0$ whenever a binary contact flag is on.

This is deliberately not the most obvious idea at this intersection. The obvious idea uses event silence as a contact anchor: no events means no slip, so velocity is clamped to zero. That works for the wrist-mounted camera in Combined Physics and Event Camera Simulator for Slip Detection, which simulates a Franka gripper with a wrist-mounted event camera so that a held object is static in the sensor frame and any rotational slip produces an event burst, classified as slip or no slip. From any camera not rigidly attached to the hand, however, a firmly held object moving through space fires events everywhere. Silence then says nothing about contact, and the binary "slip / no slip" output is exactly what we want to move past. We test the silence-anchor variant as an ablation precisely because we expect it to fail off-wrist.

Research Program

The work has three parts.

Measuring relative slip from events. Given a coarse hand mesh, an object pose estimate, and candidate contact vertices, we project each contact into the event camera and take a thin strip on each side of the finger-object boundary. In each strip, we estimate local optical flow from events over short adaptive windows. Event-based flow estimation follows EMatch, which provides unified event-based optical flow and stereo. Each window closes once enough events arrive, rather than at a fixed bin, so that fast slips get short windows and slow ones get long ones.

The flow difference across the boundary, lifted to metric units with the estimated depth, gives $\hat{s}_{k,t}$. The event count and the flow residual give $w_{k,t}$. When both strips are silent, the relative velocity is genuinely zero and receives high confidence. When one strip is occluded by the hand itself, confidence drops toward zero and the measurement does not constrain the solution.

This part draws on two lines of evidence. Event-based Vision for Early Prediction of Manipulation Actions classifies manipulation actions from event time surfaces and found that event activity marks the instant of hand-object contact; it only classifies, however, and never estimates contact velocities. Evetac puts an event camera inside an optical tactile sensor and detects incipient slip at 1 kHz, but it requires a sensor at the fingertip. We ask whether a comparable slip signal can be read from an external camera.

Slip-aware reconstruction. We add $\mathcal{L}_{slip}$ to a test-time optimization for hand and object pose. This follows the structure of ContactArt, which refines hand and articulated-object poses with a contact diffusion prior that pulls hand vertices toward object surfaces. ContactArt treats contacts as static attraction targets. We keep its attraction term but let the tangential behaviour be dictated by the measured slip. Penetration and attraction terms of the kind used by MACS are retained, so the only change is the tangential target.

Slip-labelled generation. The reconstructed sequences give per-vertex labels for three contact states, sticking, sliding with velocity, and free, rather than binary contact. We train a mass- and text-conditioned hand-object diffusion model that predicts these labels jointly with motion. A diffusion model is a generator trained to turn random noise into samples by gradual denoising. The model applies $\mathcal{L}_{slip}$ with its own predicted slip as the target. This addresses the limitation that MACS and Text2HOI share: both anchor fingers to fixed contact maps, so pivoting or sliding cannot be generated without being penalized as drift.

Data. Because external event cameras over dexterous hands with 3D ground truth do not exist in the file, we build three sources.

The first is a simulated benchmark extending the Isaac Sim plus v2e pipeline of Combined Physics and Event Camera Simulator for Slip Detection to third-person cameras. It covers tasks with scripted sticking and sliding phases: gravitational pivoting, edge sliding, and rotations of a held object, with randomized friction and textures. Exact contact velocities are available from the simulator. That paper reports 2.7 minutes of generation per sample, which bounds the benchmark at a few thousand sequences.

The second is ARCTIC, a bimanual hand-object dataset with 2.1M frames and motion-capture ground truth. We convert its videos to events with v2e and use them as a realism check on real hands. These events are interpolated from frames, so ARCTIC cannot demonstrate any benefit specific to high temporal resolution.

The third is a small real capture in which an event camera (the DAVIS346 class used by Reinold et al.) is placed beside a synchronized RGB camera. It watches a parallel gripper performing gravitational pivoting in the style of In-Hand Gravitational Pivoting Using Tactile Sensing, with object angle measured by markers, plus human hands sliding and pivoting instrumented objects.

Baselines

Naive reference: clamping. The simplest reference is the hand-object reconstruction optimization with no contact-velocity term, followed by post-hoc clamping that sets fingertip relative velocity to zero whenever a fingertip is within a distance threshold of the surface. This is the standard footskate-cleanup fix. It is run on the same pose initializations, and it defines how much slip a purely geometric rule removes. It should do well in sticking phases and fail badly in sliding phases.

Binary contact with zero velocity. This is the classical recipe of Combining Recurrent Neural Networks and Adversarial Training, transplanted to fingertips. A contact classifier predicts a binary state, and a zero-velocity loss acts during contact. We train two classifiers, one on RGB frames and one on events, with identical backbones and data. It is the strongest representative of how the motion-synthesis community handles skating.

Strongest existing contact-prior method. We run ContactArt-style test-time adaptation with its contact diffusion prior, using the same initial poses, the same optimization steps, and the same tuning budget for loss weights. On HOI4D the paper reports that the contact prior lowers hand MPJPE from 64.3 mm to 41.9 mm, and reports 35.99% object pose accuracy under the 5° 5 cm metric. On ARCTIC the reported reference is ArcticNet-LSTM at 8.9 mm motion deviation in allocentric views (ARCTIC). For our central metric, contact slip error, there is no reported result on Lacuna for any hand-object method, so we compute it for every baseline ourselves.

Physics projection. A physics-tracking stage in the spirit of LAGOON and CRISP projects kinematic motion through an RL tracking controller in a simulator. CRISP reports that tracking failure on whole-body motion falls from 55.2% to 6.9%. Here it is run in the simulated benchmark with matched controller training compute. It tests whether simulated friction alone recovers correct slip without any measurement.

Ablations. Ablations isolate the key ingredient, a measured continuous slip target. We replace $\hat{s}$ with zero wherever events are silent (the silence-anchor idea), binarize $\hat{s}$, and drop the confidence weights.

What counts as winning. Beating the baselines means two things. In sticking phases, residual contact slip should be lower than that of the strongest baseline. In sliding phases, slip velocity error should fall clearly below what any zero-target method can achieve. Both should hold without increasing MPJPE or ARCTIC motion deviation.

Why not just…?

Why not just compute slip from the reconstructed meshes themselves? Mesh-derived relative velocity is exactly the quantity contaminated by skating, so using it as a target is circular. It rewards whatever drift the estimator already has. This is still a real competitor when reconstruction is good, as with multi-view ARCTIC capture. The ablation in which $\hat{s}$ comes from the unconstrained reconstruction's own mesh velocities settles it: if that ablation matches our method, the event measurement is unnecessary and we will report so.

Why not just use high-speed RGB with optical flow at the contact boundary? This is the most serious alternative, because the measurement idea does not strictly require events. The evidence against it is indirect. Evetac produced only 1.7% of the data of a comparable frame-based tactile sensor while resolving vibrations up to 498 Hz, and frame flow degrades under blur at fast pivots, which reach 750°/s in In-Hand Gravitational Pivoting Using Tactile Sensing. We settle it directly. A frame-flow version of $\hat{s}$ is computed at several frame rates, both at matched data bandwidth and at matched frame rate, in simulation and on the real capture. If high-speed RGB matches events at equal bandwidth, the contribution reduces to the slip-target loss, and the sensor claim is dropped.

Why not just train an RGB classifier for sticking versus sliding states? A learned three-state classifier, the classifier-based variant of the binary contact recipe, could separate the two regimes without measuring velocity. However, it still provides no velocity during sliding. It is also bound by the self-occlusion at the moment of touch that ContactArt documents for small objects under the hand. We include it as a baseline and report sliding-phase velocity error separately, which is the metric on which a classifier cannot compete.

Why not just project through a physics simulator? Simulated friction can make motion plausible, but it cannot know whether a given slip was intended. That depends on intent and on unknown real friction coefficients. We expect the physics-projection baseline to reduce penetration but to misplace the onset of sliding. Its sliding-onset timing error in the simulated benchmark is the test.

Proposed Experiments

Metrics. We introduce finger skate: mean tangential relative velocity at ground-truth sticking contacts, in mm/s, the direct analogue of foot-skate metrics. We also introduce slip tracking error: velocity error at ground-truth sliding contacts, together with timing error for sliding onset. Standard metrics are reported alongside: MPJPE, ARCTIC motion deviation, three-state contact F1, and the non-collision ratio used by MACS, which reports 97.84%.

Decisive check in simulation. The first decisive check is on the simulated benchmark, where ground truth is exact. The method must reduce finger skate relative to the strongest contact-prior baseline and bring slip tracking error well below the error obtained when sliding is ignored, which equals the true slip speed. The silence-anchor ablation should collapse whenever the camera is not on the wrist. If it does not collapse, the simpler binary idea suffices and the proposal narrows accordingly.

Sim-to-real check. The second check is transfer to the real pivoting capture. Prior evidence warns of a large gap: the slip classifier of Combined Physics and Event Camera Simulator for Slip Detection fell from 80% simulated validation accuracy to 63% zero-shot on real data. Our measurement is model-based flow rather than a learned classifier, so we predict a smaller drop. We test this by reporting angle-tracking error on the gripper capture. That error is reported in degrees and is not directly comparable to the 7.23° steady-state error of RSE-LSTM with pillar tactile sensors, which used a different sensor; it is reported only for context.

Downstream generation check. The third check is whether slip-labelled generation produces pivoting and sliding that judges cannot tell from captured motion. Using the blind forced-choice protocol of MACS, we compare a generator trained on three-state labels against one trained on binary contact. We also measure finger skate and slip tracking error on the generated sequences against held-out simulated ground truth.