Hands as Rulers: Self-Calibrating Refraction from Contact and Disocclusion

When a camera looks through glass or water, light bends at the boundary, so the usual assumption that every pixel sees along a straight line from one point is wrong. The error is not a fixed image warp that can be undone once. It depends on how far each surface is from the boundary. Hand-object reconstruction methods, which estimate how a hand grasps an object and keep the two meshes from passing through each other, work at millimeter tolerances and all assume straight-line optics. The obvious fix is to measure the glass or water boundary beforehand with a checkerboard and trace rays correctly. That fix is unavailable in many of the settings where people actually manipulate things behind refractive boundaries: tanks whose water level changes, containers of unknown wall thickness, hand-held housings that shift between sessions, or rippling water surfaces. In those settings the boundary has to be estimated from the manipulation video itself.

The proposal is to use the hand as a calibration object. A human hand has roughly known metric size and anatomically bounded joints. It cannot pass through a rigid object, and where it touches the object, the surface under the fingertip must sit within a millimeter or two of the skin. That surface is hidden during the grasp and becomes visible when the finger lifts, which is a disocclusion: a previously hidden region coming into view. If the refraction model is wrong, the reconstruction breaks these facts in systematic, depth-dependent ways. Fingers end up inside the object or floating above it, bone lengths drift with depth, and the surface revealed after a finger lifts does not land where the fingertip was. We propose to treat these physical-plausibility violations as a training signal for the unknown refractive boundary, jointly with the hand, the object and the object's hidden surfaces.

Core Claim

The core claim is that physical plausibility constraints are a useful source of self-calibration for an unknown refractive interface, and that they help most in handheld sequences, where photometric multi-view consistency is weakest. The core claim is not simply that contact models need refraction-aware optics; a calibrated camera model would supply that. The gap is concrete. Refractive Structure-from-Motion Through a Flat Refractive Interface (a method that recovers camera motion through a flat interface using a generalized epipolar constraint) reports that ignoring refraction leaves a 5.24 mm mean translation error on checkerboards, cut to 1.21 mm with refractive modeling. ContactOpt (differentiable grasp refinement from predicted contact maps) is designed around only 1–2 mm of allowed soft-tissue penetration. An uncorrected refraction error is therefore larger than the entire contact tolerance. Refractive SfM, however, assumes a rigid static scene and depends on a pinhole warm start. NVS-HO (a benchmark of novel view synthesis for hand-manipulated objects) shows that handheld sequences are exactly where pose estimation from appearance collapses, because hands occlude the object and backgrounds are featureless. Photometric self-calibration is weakest in this regime, and it is also the regime where the hand supplies extra constraints. The claim can be falsified: if photometric-only refractive self-calibration already reaches contact-level accuracy on handheld sequences, the physical terms add nothing, and we will report that.

Research Program

The system reconstructs a hand-object sequence seen through a refractive interface. The interface has parameters $\theta$: distance from the camera, normal, refractive index, and wall thickness for a glass layer. Per frame, the system estimates hand pose and shape $h_t$ and object pose $T_t$, and it keeps one object shape $S$ in the object's own coordinate frame. All projection goes through piecewise-linear Snell's-law ray paths. Forward projection through a flat interface normally requires solving a high-degree polynomial. We use the pixel-wise varifocal linearization of Kawahara et al., which assigns each pixel a virtual focal length and turns this into a fast linear problem, so that projection stays differentiable and cheap inside the optimization loop. The ray tracing follows the multi-media models of Yamashita and the backward triangulation of Lin Gui et al..

The joint objective is

$$ \min_{\theta, \{h_t, T_t\}, S} \; L_{\text{photo}}(\theta, h, T, S) + \lambda_{p} L_{\text{pen}}(h, T, S) + \lambda_{c} L_{\text{con}}(h, T, S) + \lambda_{a} L_{\text{anat}}(h) + \lambda_{d} L_{\text{dis}}(\theta, h, T, S), $$

where $L_{\text{photo}}$ is the rendering and silhouette error under refractive ray paths. $L_{\text{pen}}$ penalizes hand-object interpenetration beyond a compliance band. $L_{\text{con}}$ pulls predicted contact regions together. $L_{\text{anat}}$ keeps joints inside anatomical limits and bone lengths constant across frames. $L_{\text{dis}}$ is the contact-disocclusion consistency term described below. The $\lambda$ values are scalar weights.

The physical terms reuse existing machinery rather than inventing new collision models. Penetration and contact follow ContactOpt's compliant formulation, which tolerates 1–2 mm of penetration, and the attractive and repulsive springs of CPF (a spring-mass contact model). The anatomical term uses CPF's A-MANO joint frame. The sphere-based repulsion of HO-3D_v3 and the capsule volumes of Keep It SMPL serve as cheaper interchangeable penetration terms that we will compare. These papers all supply method, and all assume pinhole optics.

Two things are new here. First, these terms are evaluated in the metric 3D space implied by $\theta$, so their gradients flow into the interface parameters and not only into the hand pose. Second, compliance is deliberately required so that we avoid the contact-penetration trade-off noted by Layout-your-3D: hard zero-penetration barriers push meshes apart until they float, and a floating hand would bias the depth estimate.

The disocclusion term $L_{\text{dis}}$ is where the three directions actually meet. While a fingertip rests on the object, the object surface under it is invisible. The contact prior says that this surface lies within the compliance band of the fingertip skin, expressed in the object frame through $T_t$. When the finger lifts at a later frame, that patch becomes visible through the interface, and refractive rendering places it at a depth that depends on $\theta$. $L_{\text{dis}}$ penalizes disagreement between the depth predicted from contact and the depth recovered after the reveal. Before the reveal, the contact prior also initializes the hidden geometry, so the hole under the fingers is filled by a bounded surface estimate instead of an unconstrained hallucination.

The hidden object geometry is represented with ray-based distances in the style of DDF-HO, which predicts hit distance and visibility along camera rays. Its authors showed this beats point-sampled signed distance fields under heavy hand occlusion. A ray-based representation also fits refraction naturally, because each bent segment is simply a new ray. DDF-HO works only in air and has no contact mechanics, and this is the part we extend.

Wavy interfaces are handled in a second stage. The flat-parameter $\theta$ is replaced by a time-varying refractive field of the kind NeRFrac uses; NeRFrac bends rays through a learned water surface to render a static submerged scene. Our hypothesis is that contact constraints stabilize the surface field in frames where the hand disrupts the unobstructed multi-view consistency NeRFrac relies on. Participating-medium effects are out of scope. The clear-water assumption is stated explicitly, and WaterSplatting's medium model, which renders underwater scenes with attenuation and backscatter but straight rays, could be added later.

Data comes from two sources. A synthetic suite renders hand-object grasp sequences, built from MANO grasps on ObMan- and HO-3D-style objects, through ray-traced flat interfaces with randomized distance, tilt, index and glass thickness, with exact ground truth for everything. A real suite captures about 30 objects manipulated inside a glass tank. It follows NVS-HO's dual-sequence protocol: a handheld sequence for reconstruction and a ChArUco board sequence for ground-truth object pose. The interface is independently calibrated with a checkerboard so it can serve as ground truth, and the setup is re-arranged between sessions (water level, tank position) so that the interface is genuinely unknown to the method.

Baselines

The naive reference point runs the standard in-air pipeline unchanged on refractive footage: pinhole projection, an off-the-shelf hand-object initializer, and ContactOpt refinement. It measures how much damage refraction does to contact reasoning. The only reported ContactOpt figure is in air, an MPJPE reduction from 57.7 mm to 48.1 mm on HO-3D in the ContactOpt paper. There is no reported result on Lacuna for any hand-object method under refraction, so this number is context rather than a target. We will also run this baseline on in-air control captures of the same objects to separate the cost of refraction from the cost of our capture setup.

The strongest competitor that uses no physics is photometric-only refractive self-calibration. It is our full system with $L_{\text{pen}}$, $L_{\text{con}}$, $L_{\text{anat}}$ and $L_{\text{dis}}$ removed, which amounts to the RSfM recipe of Chadebecq et al. (pinhole warm start, then refractive reprojection refinement) applied to the object-relative motion with hand pixels masked. Its reported accuracy on checkerboards, 1.21 mm mean translation error and 0.35° rotation error, is the level it reaches on ideal targets. The question is how far it degrades on handheld, occluded, low-texture sequences. It gets the same initialization, the same refractive projection code, the same iteration budget and a hyperparameter sweep of equal size to ours. Beating it means lower interface-parameter error and lower object translation error on the real suite, together with a reduction in penetration depth and intersection volume larger than the variation across random seeds.

The oracle upper bound is the full system with $\theta$ fixed to the checkerboard calibration. We succeed if our uncalibrated system closes most of the gap between the photometric-only baseline and this oracle.

Ablations isolate the key ingredients. We drop $L_{\text{dis}}$ alone to test whether contact-disocclusion consistency matters beyond static contact. We drop $L_{\text{anat}}$ alone to test whether the hand's metric prior is doing the calibrating. We swap the compliant penetration term for a hard barrier to test whether floating biases the interface estimate.

Why not just…?

Why not just calibrate the interface once with a checkerboard? When the interface is fixed and accessible, this is the right answer, and we include it as the oracle rather than argue against it. The proposal matters only when calibration goes stale or is impossible: a moved tank, a changed water level, an unknown container, a housing that flexes. The real captures deliberately re-arrange the setup between sessions to reproduce this. If the oracle and a single stale calibration perform the same, the problem is less pressing than we think, and the experiment will show that.

Why not just undistort the images in 2D and then run existing hand-object tools? Refractive displacement depends on each point's distance from the interface, as Chang et al. emphasize, so no single image warp is correct for both a fingertip and the object surface behind it. Refractive Structure-from-Motion on Underwater Images shows that straight-ray pipelines bend flat surfaces and drift in trajectory. We will run a best-case version of this baseline anyway, a warp fitted using the ground-truth interface at the mean object depth, to measure how much error remains at contact scale.

Why not rely on photometric refractive self-calibration alone? This is the real threat. If handheld sequences contain enough texture, RSfM-style estimation may already recover $\theta$ well, and the physical terms would only polish the hand pose. The photometric-only baseline tests this directly. We expect it to degrade on low-texture objects and heavily occluded sequences, based on NVS-HO's reports of pose collapse, and we will stratify results by object texture and occlusion fraction. If there is no gain in any stratum, the core claim fails.

Why not let a diffusion prior fill the disoccluded regions and skip physics? FreeFix, which supervises 3D Gaussian splatting with 2D diffusion priors at wide viewing angles, notes that such priors invent structure that is inconsistent across views. Nothing in them ties the region under a finger to the finger's surface. We will compare diffusion-based filling against contact-anchored filling on the ground-truth geometry of revealed patches in the synthetic suite, where the hidden surface is known exactly.

Proposed Experiments

The first decisive check is a synthetic sweep over interface tilt and distance. For each setting we plot interface-parameter error and object translation error for the photometric-only baseline, our system, and the oracle, against the fraction of the object hidden by the hand. The prediction is that the advantage of physics-guided calibration grows with occlusion. If the curves stay flat, the hypothesis is refuted.

The second check is on the real tank suite. We report object pose error against the ChArUco board sequences, hand MPJPE against a multi-view in-air capture of the same grasps, penetration depth and solid intersection volume as used in CPF's evaluation on the First-Person Hand Action benchmark, and a gravity-stability test in Bullet following CPF.

The third check measures disocclusion quality. We compute Chamfer distance on revealed patches in the synthetic suite, and masked PSNR and LPIPS restricted to disoccluded pixels in held-out real views. This tests whether contact-anchored completion beats both the diffusion alternative and completion without the contact prior.

The fourth check tests whether the gains carry over to a rippling surface, using NeRFrac's synthetic ripple setting with an inserted hand-object interaction. For reference, NeRFrac itself reports 34.98 PSNR on the unoccluded "second sine" scene against 25.93 for vanilla NeRF. We will state whether adding contact constraints stabilizes the water-surface estimate during frames where the hand occludes the scene.