The Shadow as a Second Camera: Tracking Occluded Fingers from Their Cast Shadows

When a person grabs a mug on a table, a head-mounted camera often loses sight of the fingers wrapped around the far side. Yet a lamp overhead is still "seeing" those fingers and drawing their outline on the table as a shadow. We propose to treat that shadow as a free second viewpoint. A single-camera hand tracker would explain both the hand pixels and the hand's cast shadow, given an estimated light direction. A learned motion predictor would carry the hand's pose through frames where both cues fail. The same light estimate is then reused to render virtual objects whose shadows fall consistently on the real hand and the real table.

This matters because bare-hand direct manipulation is the interaction style that interaction-design studies favor: users find grabbing, colliding and moving objects with their own fingers immediately intuitive (Winkler et al., In et al.). It is also the style that breaks vision trackers. DexYCB reports that object pose accuracy falls from about 56% to 41% once the object is grasped, and that RGB-only absolute hand-joint error is 52.2 mm against 27.5 mm for depth-based methods. The standard fix is tactile gloves (ViTaM-D), which removes the bare hand from the interaction. A geometric signal that survives grasp occlusion and needs no extra hardware would address the main reason consumer AR still falls back on ray-casting and buttons.

Core Claim

Under any scene lighting with a dominant direction, the hand's cast shadow on the support surface carries information about finger configuration that the camera cannot see during grasp. That information can be used explicitly. The tracker estimates the light from the shadow of the known, tracked object. It then scores each hypothesized hand pose by how well the rendered joint hand-plus-object shadow matches the observed shadow region. We expect this to substantially lower joint error on camera-occluded fingers compared with the same tracker without the shadow term. The gain should shrink smoothly to zero as lighting becomes diffuse, and the system should not get worse when shadows are absent.

The four research directions play different roles here. Occlusion-robust predictive tracking is the core mechanism. Global illumination supplies a new observation channel, and the light estimate it produces is shared between tracking and rendering. Hybrid motion modeling supplies the hand surface whose shadow is rendered. Intuitive gesture mapping is the application regime and the user-facing evaluation.

What Exists and What Is Different

Two papers use hand shadows, and both stop short of what we propose. Mask2Hand regresses MANO pose and shape (MANO is the standard parametric hand mesh with 778 vertices) from a binary silhouette, refined by vertex offsets and a differentiable contour loss. It reaches 3.56 cm unaligned joint error on FreiHAND from silhouettes alone, against 4.31 cm for an RGB baseline restricted to binary input. Its "shadow" is the hand's own mask in the camera view, it works per frame, it handles isolated hands, and its authors name temporal information as the missing piece for depth ambiguity. Hand-Shadow Poser inverts target shadow art into two-hand poses with diffusion, a transformer and differentiable rendering. It assumes a fixed known light, handles static frames, and takes 56 seconds per pose.

These two papers motivate the proposal by showing that silhouettes constrain articulated pose. We differ in using a real cast shadow on scene geometry, seen from a different direction than the camera. That shadow is a complementary view rather than the same view stripped of color. We also estimate the light online and run the shadow term inside a temporal tracker at interactive rates.

The temporal machinery is borrowed rather than invented. MambaTrack shows that a bidirectional state space model (a sequence model with a recurrent hidden state that scales linearly in sequence length) can carry non-linear trajectories through missed detections. It reaches 56.8 HOTA on DanceTrack at 11.37 ms per frame, but only for boxes. DF-Mamba brings deformable state space blocks to interacting-hand pose and reports 7.94 / 10.53 / 9.32 mm MPJPE (single / two hands / all) on InterHand2.6M. It ignores lighting and outputs joints only. We use a DF-Mamba-style predictor as the motion prior and add a physically rendered observation it lacks.

On the representation side, XHand subdivides MANO and predicts pose-dependent offsets. This brings point-to-surface error from 5.659 mm to 2.276 mm at 56.2 FPS, but XHand assumes clean pose input and fixed lighting. NCRF jointly refines hand-object pose with a radiance field. It explicitly does not model cast shadows or temporal filtering. Structured-NeRF shows that adding shadow mapping is what makes inserted objects stop looking like floating holograms (3.88 vs 1.90 out of 5 in its user study). It handles only static rigid scenes. Each of these treats shadows as something to render or ignore. None treats them as something to track with.

Research Program

The system has four parts.

The first part estimates the light, using the object as a probe. In tabletop manipulation the grasped object usually has a known mesh, as do the YCB objects in DexYCB, and is tracked in 6D. We fit a dominant directional or point light, plus an ambient term, by matching the object's rendered shadow on the estimated table plane to the detected shadow region. This uses frames before contact, when the object is unoccluded. The light is then held fixed with slow updates. The procedure needs no light probe and no extra hardware.

The second part segments shadows. We will compare two detectors for shadow pixels on the support surface. One is a ratio image against a running background model of the table. The other is a small learned shadow segmenter trained on synthetic renders. Both produce a soft shadow mask $S_t$ at frame $t$. Hand and object shadows typically merge during grasp. We therefore never try to separate them: the model renders their joint shadow.

The third part is the tracker. The state $x_t$ is MANO pose and shape plus object 6D pose. A DF-Mamba-style predictor trained on hand-object sequences proposes $\hat{x}_t$ from the history. The update step then runs a few gradient steps on

$$E(x_t) = E_{\text{img}}(x_t) + \lambda_s \, w_t \, D\big(R_\ell(x_t), S_t\big) + \lambda_p \, \lVert x_t - \hat{x}_t \rVert^2 .$$

Here $E_{\text{img}}$ is the usual 2D keypoint and mask term in the camera view. $R_\ell(x_t)$ is the differentiable shadow silhouette of hand and object cast by light $\ell$ onto the table plane. $D$ is a distance-field contour loss of the kind Mask2Hand uses. $w_t$ is a confidence weight set by the measured shadow contrast. The last term keeps the solution near the prediction. The weight $w_t$ is what lets the method fall back to the plain tracker under diffuse light.

Shadow rasterization of a hand mesh from one extra viewpoint costs about as much as one more silhouette render. Our target is 30 FPS end to end on a single GPU, with the predictor's per-frame latency comparable to MambaTrack's 11.37 ms. The shadow silhouette is rendered from MANO, or from the XHand subdivided surface in an ablation that tests whether finer geometry matters for thin fingers.

The fourth part closes the rendering loop. The estimated light and the tracked hybrid hand drive an AR renderer. Virtual objects receive shading from $\ell$, cast shadows onto the real table and the tracked hand, and the hand casts shadows onto virtual objects. This is the bidirectional transport that the illumination literature names as missing for moving objects. Here it comes almost for free because the tracker already needs $\ell$.

Baselines

The naive reference is the single-frame pose estimator with no temporal model and no shadow term. Concretely, we use the per-frame backbone of our tracker, which shares DF-Mamba's architecture, applied independently to each frame. It shows how bad grasp occlusion is in our data, and it should reproduce the qualitative interaction penalty DexYCB documents.

The classical temporal reference is the same per-frame estimator followed by a constant-velocity Kalman filter on MANO parameters. This is the textbook gap-bridging approach (Gennery, Zorina et al.).

The strongest existing method is DF-Mamba, retrained on our training splits with temporal context. Its reported 9.32 mm overall MPJPE on InterHand2.6M is the reference number. For DexYCB we found no reported DF-Mamba number on Lacuna, so we will reproduce it.

Fair comparison means identical training data, input resolution, single-camera input and an equal hyperparameter search budget. All methods except the naive one also get the same object pose tracker. Beating DF-Mamba means a statistically significant reduction in MPJPE restricted to camera-occluded fingertips under directional light, with no regression on unoccluded joints or under diffuse light.

The decisive ablation is our full system with $\lambda_s = 0$, which isolates the shadow term from everything else. A second ablation replaces the estimated light with the ground-truth light, to separate light-estimation error from the value of the shadow cue itself.

Why not just…?

Why not just add a second camera? A second calibrated camera beside the lamp would see everything the shadow shows, plus color and depth. It is a real upper bound, and we will run it as one. The argument for shadows is deployment. Head-worn devices have one forward-facing viewpoint, and the light is already in the room. The experiment reports the fraction of the gap between single-view and two-view tracking that the shadow closes. If that fraction is small, the idea is not worth its complexity.

Why not just let a big network see the shadows implicitly? This is the most serious threat. A sequence model given the full frame may already exploit cast shadows without any explicit renderer. We test it directly. We evaluate trained DF-Mamba on test frames where the shadow region has been inpainted away; if accuracy does not change, the network is not using shadows. We also compare the explicit shadow term against simply giving DF-Mamba the shadow mask as an extra input channel. If the extra channel matches the renderer, we will report that the cue matters and the explicit geometry does not.

Why not just use a stronger motion prior or a Kalman filter? Prediction alone cannot recover finger changes that happen while the fingers are hidden, such as tightening a grip or sliding a finger behind the object. That is exactly the regime of abrupt stops on contact that the tracking literature flags as breaking linear models. MambaTrack's roughly 10-point HOTA gain over the Kalman-based ByteTrack on SportsMOT shows the learned prior helps. The $\lambda_s = 0$ ablation shows whether a learned prior is enough, broken down by occlusion length.

Why not just use depth? DexYCB's 27.5 mm versus 52.2 mm gap shows that depth helps. But a head-mounted depth sensor shares the camera's viewpoint and is blocked by the same object, so depth does not solve the far-side-finger problem. We will include an RGB-D variant of the baseline where depth is available, and also check that the shadow term is additive to depth.

Proposed Experiments

Existing benchmarks were not captured for strong shadows, so the evaluation has three tiers.

The first tier is synthetic. We re-render DexYCB and HO3D hand-object sequences with ground-truth MANO and object poses. We sweep light direction and softness from a hard point light to fully diffuse, and choose a camera that produces controlled occlusion. This yields MPJPE on occluded joints as a function of shadow contrast. The predicted curve is gain proportional to contrast, near zero under diffuse light, never negative.

The second tier is real existing data. We take DexYCB and HO3D with one camera as input and the remaining DexYCB views providing ground truth. We first measure how much shadow contrast these captures actually contain and report results stratified by it. We also confirm on InterHand2.6M that the tracker without shadows reproduces DF-Mamba-level accuracy, so improvements are not baseline artifacts.

The third tier is a new capture: a tabletop rig with one dominant lamp, a head-mounted RGB camera, and a multi-camera ring for ground-truth triangulation. Subjects perform grasp, rotate and place tasks with YCB objects. This is the setting the claim is about.

Metrics are as follows. Tracking is measured by MPJPE split into visible and camera-occluded joints, by track-loss rate during grasp, and by object pose average recall. Plausibility is measured by penetration depth. Runtime is measured in FPS. Rendering is evaluated with a blind preference study of AR compositions using our estimated light against fixed-light rendering, in the spirit of Structured-NeRF's user study.

A short interaction study then has participants perform direct-manipulation tasks: grab, collide-to-merge and throw-to-delete, following In et al.. Each participant uses both the full system and the $\lambda_s = 0$ system, and we record task success and tracking dropouts. This tests whether lower joint error turns into fewer broken grasps.

The decisive check is the $\lambda_s = 0$ comparison on camera-occluded fingertips in the lamp-lit capture, read alongside the shadow-inpainting test of DF-Mamba. If the explicit shadow term wins there, holds at 30 FPS, and causes no regression under diffuse light, the claim stands. If an implicitly trained network matches it, the contribution shrinks to a dataset and an analysis showing that shadows carry the signal.