Single-image 3D hand reconstruction keeps failing in the same place: fingers hidden behind other fingers, the palm, or a held object. A natural fix is to feed the network a depth map from a pretrained monocular depth estimator, a network that guesses per-pixel distance from one photo. The catch is that at a hidden finger's image location, the depth map does not report the finger. It reports the surface in front of it. Systems that treat that number as the finger's depth pull hidden parts forward onto the occluding surface. Systems that learn to ignore it throw away a useful fact: the hidden finger must lie behind that surface.
We propose to use monocular depth according to visibility. For hand mesh vertices that are visible, depth is an equality target. For hidden vertices, it is a one-sided bound. Which vertices are visible is decided by a coarse skeleton stage, before any fine mesh refinement. The same visibility-aware depth score can also rank multiple sampled pose hypotheses. That lets a probabilistic coarse-to-fine model output one answer instead of relying on an oracle "best of N" pick.
Core Claim
The research file's own gap analysis observes that monocular depth models predict the depth of the occluding foreground rather than the occluded part. It also notes that no existing system pairs an explicit coarse visibility estimate with selective use of the depth prior. The closest systems handle depth in one of two ways. Depth-Guided Metric-Aware Temporal Consistency fuses Depth Anything v2 features with RGB features through learned confidence gating. It works on video and regresses body-model parameters, with no per-vertex visibility. Cascaded Diffusion for coarse-to-fine hand pose samples several 3D skeletons, then refines each into a mesh, but uses no depth at all.
Our claim is that the right operation on depth under occlusion is neither gating nor dropping it, but switching the constraint type per vertex. The visibility mask that decides this switch can be computed cheaply from coarse skeleton hypotheses. We take the occlusion-aware principle from Cuboids Revisited, which penalizes cuboids that would sit in front of observed surfaces. There, removing the penalty raised mean fitting error from roughly 20 cm to over 1.5 m on NYU Depth v2. We apply that principle to articulated hands, where the occluder is often the hand itself.
Research Program
The model has three stages. First, a coarse stage produces $K$ 3D joint skeleton hypotheses from the image, in the style of the cascaded diffusion model. Second, each hypothesis is turned into a coarse hand mesh, using the MANO hand template posed by the joints. That mesh is soft-rasterized to give a per-vertex visibility weight $w_v \in [0,1]$. Third, a fine stage refines vertex positions, conditioned on RGB features, Depth Anything v2 features, and the visibility map.
The depth prior enters through one energy, used both as a training loss and as a test-time score:
$$E = \sum_v \Big[ w_v \,\rho\big(z_v - \hat d(\pi(v))\big) + (1-w_v)\,\max\big(0,\ \hat d(\pi(v)) + \delta - z_v\big) \Big]$$
The symbols are as follows:
- $z_v$ is the camera-space depth of vertex $v$, and $\pi(v)$ is its image projection.
- $\hat d$ is the monocular depth after a per-image scale-and-shift alignment. The alignment is fitted only on vertices currently judged visible, following MonoSDF's handling of scale-ambiguous depth.
- $\rho$ is a robust penalty.
- $\delta$ is a small thickness margin.
Pixels within a thin band around depth discontinuities are down-weighted. This is because the file documents edge blur in monocular depth near overlapping boundaries (Few-Shot Novel View Synthesis Using Depth Aware 3DGS), and those boundaries are exactly where visibility flips.
The same energy covers occlusion by objects. A hand behind a mug must also be behind the mug's depth, so hand-object datasets test the idea without special casing.
During training, visibility is first computed from ground-truth MANO meshes by z-buffering, then switched to visibility predicted from the coarse hypotheses. This exposes the fine stage to its own visibility errors.
At test time, all $K$ hypotheses are refined. The one with the lowest $E$, plus the 2D keypoint reprojection term, is returned as a single answer.
The analysis targets two questions. Does the inequality term reduce hidden-joint error specifically? Does $E$ order the hypotheses well enough to close part of the gap between single-output and best-of-N accuracy?
Baselines
The naive reference is the strongest RGB-only single-shot regressor we can reproduce. HandOccNet splits features into visible and occluded slots and injects context into the occluded slots. It reports 9.1 mm mean joint error on HO-3D and 10.8 mm on FPHA. HandGCAT fuses graph-encoded 2D skeleton priors into occluded image regions. It reports 8.7 mm PA-MPJPE on HO3D v2 and 5.60 mm on DexYCB. These define the bar for "depth adds nothing".
The strongest existing method for this question is Cascaded Diffusion. It reports 7.4 mm P-MPJPE on HO3Dv2 and 4.4 mm P-MPJPE on FreiHAND. The FreiHAND figure is explicitly Best-of-N, an oracle pick. Our single-output result must be compared against its single-output mode, which we will run ourselves under the same sample count $K$. Its Best-of-N number is reported separately as an upper reference.
The depth-fusion competitor is the confidence-gated Depth Anything v2 fusion of Depth-Guided Metric-Aware Temporal Consistency, adapted to single images and inserted into the same cascaded backbone. It reports 69.31 mm MPJPE on 3DPW, but we found no reported result on Lacuna for it on any hand benchmark, so we will measure it. It is the baseline we most need to beat: it already uses the same depth features and could, in principle, learn to suppress them where hands are hidden.
All methods share the following:
- the same image backbone,
- the same depth model,
- the same training splits,
- the same number of epochs,
- an equal hyperparameter search budget.
Three ablations isolate the key ingredient:
- equality-only depth loss on all vertices,
- depth loss on visible vertices only, with no inequality term,
- ground-truth visibility in place of predicted visibility, as an upper bound on what better visibility would buy.
Beating the gated-fusion baseline counts as success only under a specific condition. Its overall PA-MPJPE gain must hold at matched compute, and the hidden-joint error must improve by more than the run-to-run spread over three seeds.
Why not just…?
Why not just gate the depth features and let the network learn when to distrust them? This is the most serious threat. The gated fusion in Depth-Guided Metric-Aware Temporal Consistency already showed that gated intermediate features beat raw depth. Gating, however, can only scale evidence down. It has no way to express "the finger is somewhere behind this value", so on hidden joints it should behave like the visible-only ablation. The visible-only versus full-energy comparison, measured on hidden joints, settles this directly. If gating matches our method there, the inequality term is not earning its place, and we will report that.
Why not just use depth to rerank diffusion samples, with no change to training? This is cheap and could capture much of the benefit. We include it explicitly as a variant: Cascaded Diffusion samples scored by $E$ at test time only. If it matches the full model, the contribution reduces to a selection rule. That would still be useful, and we would say so.
Why not just regress MANO directly from a large image transformer? The cascaded diffusion work documents that single-shot deterministic regression averages over plausible hidden-finger poses. Scale drift and limb flips in RGB-only regressors are documented in the depth-guided human mesh work. The HandOccNet and HandGCAT reference rows quantify where this approach stands.
Why not just predict a front/back order for each joint pair? An order head gives a discrete relation but not a metric bound tied to the observed surface. We will train a pairwise-order head on the same backbone as a further baseline, scored on the ordering metric below.
Proposed Experiments
Training and evaluation use HO3D v2, DexYCB, and FreiHAND, with FPHA as a cross-dataset test. Per-vertex visibility ground truth comes from z-buffering the annotated MANO meshes. Test frames are stratified by the fraction of hand vertices that are hidden, and errors are reported per stratum. The core prediction is that our gains grow with occlusion and vanish on mostly visible hands.
Metrics are:
- PA-MPJPE and mesh vertex error;
- hidden-joint PA-MPJPE;
- pairwise depth-ordering accuracy, the fraction of joint pairs whose front/back relation matches ground truth;
- for the selection rule, the gap between the selected hypothesis and the Best-of-N oracle.
The decisive check is the hidden-joint error and ordering accuracy of the full energy against the visible-only and gated-fusion variants in the most occluded stratum. If the inequality term does not improve ordering there, the central claim fails.
A secondary check tests the dependence on the depth prior directly. We corrupt $\hat d$ with boundary blur and with scale error, and measure how fast each method degrades. A method that uses depth as a bound should degrade more gracefully than one that uses it as a target.
References
- Depth-Guided Metric-Aware Temporal Consistency for Monocular Video Human Mesh Recovery
- Cascaded Diffusion Framework for Probabilistic Coarse-to-Fine Hand Pose Estimation
- Cuboids Revisited: Learning Robust 3D Shape Fitting to Single RGB Images
- MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
- Few-Shot Novel View Synthesis Using Depth Aware 3D Gaussian Splatting
- HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network
- HandGCAT: Occlusion-Robust 3D Hand Mesh Reconstruction from Monocular Images