When a hand moves quickly while holding a cup, a video frame shows both the fingers and the cup smeared across the image. Motion blur is the record of how things moved while the camera shutter was open. Current methods either throw that record away, by deblurring first or masking blurred pixels, or they explain it as the motion of a single body moving freely in empty space. We propose to read hand and object blur jointly, under one physical fact: while fingers grip an object, they move with it. We represent the motion during the exposure as two coupled paths, a hand path from a parametric hand model and a rigid path for the object. Along sampled sub-instants of the exposure we enforce two conditions: the hand does not pass through the object, beyond a small tolerance for skin compression, and contacting finger points travel with the object surface. The rendered sub-frames are averaged to reproduce the observed blur. The object usually has a simple, well-constrained rigid streak, so it becomes evidence about where the fingers went, at the very pixels where finger evidence is worst.
This matters because fast manipulation produces the hardest frames for 3D hand-object estimation. Grasped-object pose accuracy already falls from 56% to 41% on DexYCB without blur. The contact interface, where blur and occlusion pile up, is also where plausibility methods most need image evidence.
Core Claim
Contact physics should constrain the motion during the exposure, not only the pose shown in a frame. The claim is that coupling hand and object paths through contact makes blur informative about the fingers. Its measurable consequence is lower hand joint error and lower penetration on blurred hand-object frames than (a) deblurring followed by a static contact optimizer and (b) a blur-forward model whose hand and object paths are independent. The gain should grow with blur length. That growth is the signature that distinguishes the mechanism from generic regularization.
Relation to Prior Work
Two papers supply the blur-formation machinery, and both leave the gap open. Deblur-Avatar models articulated human blur with a parametric body model (SMPL). It predicts body poses at the start and end of the exposure, interpolates intermediate poses with spherical linear interpolation, renders each one, and averages the results. Its body is alone in free space, so the interpolated poses can pass through anything. DynaMoDe-NeRF relates 3D rigid-object velocity and depth to 2D blur kernels and recovers 3D trajectories. It is limited to rigid objects with pre-segmented masks and has no notion of a second body. We reuse the forward "average the sub-frames" model from the first paper and the rigid-velocity view from the second. What is new is a second body, with physical coupling between the two.
Four papers supply the physical terms. HO-3D_v3 represents hand and object interiors as spheres and adds a cheap repulsion penalty. It also aligns joint axes with anatomy, which lowered annotation error from 12.31 mm to 8.12 mm. ContactOpt shows that allowing up to about 2 mm of penetration, to mimic soft skin, stabilizes optimization. CPF balances attractive and repulsive springs and restricts finger joints to anatomically valid rotations. All three treat isolated sharp frames. We apply their terms at sub-exposure instants and add a velocity term between them.
WildGHand motivates the proposal by contrast. It handles blur and hand-object interaction as perturbations: a small network absorbs them as transient biases, and their pixels are down-weighted. That yields a clean hand avatar, but it removes the evidence at the contact interface.
Research Program
For each blurred frame we estimate a hand path and an object path over the exposure. The hand path uses the anatomically constrained hand model described in CPF: start and end poses plus one midpoint control, so curved motion is allowed in the spirit of the quadratic paths of Exposure Trajectory Recovery. The object path is a rigid 6-degree-of-freedom path parameterized the same way. The object mesh is known, as in HO-3D and DexYCB.
At $S$ sub-instants $t_s$ we render both bodies with a differentiable renderer and average the sub-frames to form a predicted blurry image. The fit is scored by photometric error and by a blurred-silhouette error against segmentation.
The objective adds three physical terms evaluated at every $t_s$. First is the sphere-based repulsion of HO-3D_v3, with a 2 mm dead zone following ContactOpt. Second is a contact attraction for fingertip vertices predicted to be in contact. Third, and central to the proposal, is a coupling term:
$$E_{couple}=\sum_s\sum_{i\in C}\big\|\dot{x}_i(t_s)-v_O(x_i(t_s),t_s)\big\|^2,$$
Here $C$ is the set of hand vertices in contact, $\dot{x}_i$ is the hand vertex velocity along its path, and $v_O(x,t)$ is the rigid object's velocity field at point $x$. The coupling is applied softly, so sliding contact is penalized rather than forbidden.
Contact labels come from a contact predictor trained on ContactPose, as in ContactOpt. Initialization uses any per-frame hand-object estimator, and neighboring frames are tied together by continuity of the end and start poses. This continuity also settles the forward-versus-backward ambiguity of a single blurry frame.
Data
No public hand-object dataset provides blurred frames with 3D ground truth, so we build two benchmarks. The first is synthetic and follows ObMan. We render grasp sequences at high frame rate with known hand and object trajectories and average the frames into exposures of increasing length. This is the same averaging principle used to build the GoPro deblurring set. Because the true motion during each exposure is known, this benchmark measures trajectory error directly.
The second is semi-real. We take HO-3D_v3 and DexYCB sequences, interpolate frames, and average them into blurred frames, keeping the original 3D annotations as targets for the middle instant. Interpolation artifacts are a real caveat, so we report this benchmark separately. The motion-blur split of HWP serves as an in-the-wild check: it reports only appearance metrics, plus a perceptual study modeled on ContactOpt's.
Baselines
The naive reference applies a standard per-frame hand-object estimator directly to blurry frames, followed by ContactOpt refinement. This is the strongest existing static pipeline. On sharp HO-3D, ContactOpt reports MPJPE (mean per-joint position error) falling from 57.7 mm to 48.1 mm. No result on blurred hand-object frames was found on Lacuna, so we establish one.
The strongest deblur-first competitor runs Exposure Trajectory Recovery, which reports 31.05 dB PSNR on GoPro. Its deblurred middle frame is fed to the same estimator and ContactOpt. It is the strongest because it already models curved motion during the exposure and is the best deblurrer in the file.
The masking competitor reimplements WildGHand's down-weighting of perturbed pixels inside our fitting loop. It isolates whether simply ignoring blurred pixels is enough.
Three ablations isolate the proposal's ingredients. A two-body extension of Deblur-Avatar uses paths and blur averaging identical to ours, with no physical terms. A second version adds penetration and contact terms only at the exposure endpoints. A third uses all physical terms except $E_{couple}$.
For fairness, all methods share the same initializer, object meshes, contact predictor, renderer and iteration budget. Every weight is tuned on the same held-out validation split with an equal number of trials. Beating the field means lower MPJPE and object pose error than the deblur-first pipeline at every blur length, with penetration depth and intersection volume no worse than ContactOpt on sharp frames. The improvement over the endpoint-only and no-coupling ablations must widen as blur length grows.
Why not just…?
Why not just deblur first and then run a contact optimizer? Deblur-Avatar reports that 2D pre-processing introduces temporally inconsistent, hallucinated detail that corrupts 3D fitting. A 2D network also has no reason to keep the finger-object boundary geometrically faithful where both are smeared together. This is nonetheless a serious competitor on mild blur, and our blur-length sweep will show where, if anywhere, it stops being competitive.
Why not just mask the blurred pixels, as WildGHand does? The masked pixels are the contact interface, so masking removes the gradient that places the fingers on the surface. What remains is the static prior, which leaves the floating-versus-clipping trade-off that CPF and ContactOpt describe. The masking baseline tests this directly. If it matches us, blur carries less finger information than we claim.
Why not just enforce penetration and contact at the frame poses and keep plain interpolation in between? This is the real threat. If exposures are short relative to finger thickness, endpoint constraints may already prevent most mid-exposure penetration. On the synthetic data we will measure how often endpoints are collision-free while mid-exposure samples penetrate. We will also compare endpoint-only against full sampling. If the gap is negligible, we drop dense sampling and attribute any gain to the coupling term alone, which the no-coupling ablation measures separately.
Why not just train an end-to-end network from blurry frames to plausible poses? ContactOpt found explicit contact optimization superior to its learned refinement network on sharp grasps. A learned model would also need blurred 3D training data that does not exist. Our synthetic set could train one, and we will report such a network as an additional baseline if budget allows.
Decisive Checks
On synthetic data with known motion during the exposure, the full model must recover hand trajectories at mid-exposure samples better than the no-coupling ablation. The improvement must be concentrated on fingers in contact, not on free fingers. On HO-3D-derived blur, MPJPE and object pose error must improve over the deblur-first pipeline. Penetration depth and intersection volume, measured as in CPF, must not regress, and the contact ratio must not drop toward floating.
A falsifying outcome would be equal gains on contacting and non-contacting fingers, which would indicate generic smoothing rather than coupling. Another would be a vanishing gap at long blur, which would mean the object streak does not in fact inform the fingers.
References
- DexYCB: A Benchmark for Capturing Hand Grasping of Objects
- Deblur-Avatar: Animatable Avatars from Motion-Blurred Monocular Videos
- DynaMoDe-NeRF: Motion-aware Deblurring Neural Radiance Field for Dynamic Scenes
- HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset
- ContactOpt: Optimizing Contact to Improve Grasps
- CPF: Learning a Contact Potential Field to Model the Hand-object Interaction
- WildGHand: Learning Anti-Perturbation Gaussian Hand Avatars from Monocular In-the-Wild Videos
- Exposure Trajectory Recovery From Motion Blur