When a person turns an unfamiliar object in their hands in front of one camera, much of the object is never seen. Fingers cover it, and its back faces away from the lens. Current methods fill these gaps in one of two ways. They borrow a shape guess from an image generator, or they let the 3D model grow whatever geometry makes the rendered frames match. The second option is worse than it sounds. When the estimated object pose is slightly wrong, the hidden region is where the optimizer quietly puts fake geometry to cover the mismatch. Hidden geometry and wrong poses feed each other.
This proposal treats the hand differently. Methods today either mask the hand out as a nuisance or use it only as a local contact cue. Here the hand becomes a measuring instrument. We have a reliable parametric hand model with known volume, so every place a finger occupies at any moment, expressed in the object's own coordinate frame, is space the rigid object cannot fill. Over a video, the fingers sweep through exactly the regions the camera cannot see, such as behind, underneath, and between grasped parts. The idea is to turn that sweep, together with camera-ray "free space" and contact, into a partition of the object's space into three regions: known-occupied, known-empty and unknown. Image-matching gradients are then routed so they can no longer move geometry into the unknown region to absorb pose error. The object's per-frame pose is refined jointly, and it receives gradients from both the image and this geometric partition.
Core Claim
The claim has two parts. First, in hand-held reconstruction of unknown objects, the hand-model template can constrain the template-free object in the very regions where both the camera and generative priors are least reliable. Second, the benefit depends on how gradients are routed, not only on the loss terms. If image-matching error can only change appearance, and object geometry can only move under the geometric partition, then pose error has nowhere to hide.
Let $T_t$ be the object's rigid pose at frame $t$, mapping object coordinates to camera coordinates. Let $H_t$ be the 3D volume of the fitted hand at frame $t$. The swept free set in the object frame is
$$ F^{\text{hand}} = \bigcup_t T_t^{-1}(H_t), $$
which depends on the poses $T_t$. It is therefore differentiable with respect to them through a soft occupancy of the hand. The object frame is split into three regions. $\Omega_{\text{free}}$ is the union of $F^{\text{hand}}$ and the space crossed by camera rays that hit background. Hand-occluded pixels are excluded from that ray carving by rendering the current hand depth, not by trusting a 2D segmenter. $\Omega_{\text{occ}}$ is the set of observed surface points plus predicted contact points. $\Omega_{\text{unk}}$ is everything else. Training minimizes
$$ \mathcal{L} = \mathcal{L}_{\text{photo}}(\theta_{\text{app}}, T) + \lambda\, E_{\text{geo}}(\mu, T;\, \Omega_{\text{occ}}, \Omega_{\text{free}}, \Omega_{\text{unk}}), $$
where $\theta_{\text{app}}$ are the appearance, opacity and scale parameters of the object's 3D Gaussians and $\mu$ are their positions. The key restriction is that $\mathcal{L}_{\text{photo}}$ does not update $\mu$.
The gradient-routing idea is borrowed from EnerGS. EnerGS splits space into occupied, free and unknown regions using LiDAR, and lets image gradients update only appearance, scale and opacity. It assumes a static scene and fixed, calibrated cameras, and its free-space carving breaks when objects move through sensor rays. We change three things. The partition is built in a moving object's own frame. The "sensor" that carves hidden space is the hand rather than a laser. The poses that define the partition are themselves being optimized.
Research Program
The system is built on the monocular hand-object setting of HOLD. HOLD couples an articulated parametric hand model (MANO) with an unconstrained neural surface for the object and uses contact to resolve depth. It initializes object poses from classical structure-from-motion, which the HOLD authors report fails on thin or textureless objects. In our system the object is represented by 3D Gaussians, which are explicit blobs that a fast rasterizer renders into images. The per-frame object poses are refined with analytic pose gradients in the style of Look Gauss, No Pose. That method refines camera poses inside the Gaussian rasterizer, but it assumes a static world and needs an anisotropy penalty to stop Gaussians from stretching into needles that cover pose errors. We expect gradient routing to make that penalty largely unnecessary, and we will test this directly.
Optimization alternates between two steps. In the first, the hand and object poses are refined while geometry is held to the current partition. In the second, the partition is rebuilt from the updated poses and Gaussian positions take an energy step. The energy follows EnerGS: a robust attraction term in the occupied region, a repulsive barrier in the free region, and a weak prior in the unknown region.
Contact plays two roles. The first is to supply occupied anchors, in the spirit of the spring-like attraction and repulsion of CPF. CPF requires a CAD model of the object; we do not. The second is to pin $T_t$ during frames when the object's visible texture says little.
The unknown region is filled under strict limits. Optionally, a diffusion prior via score distillation (SDS) is applied as in BIGS. It is applied only to Gaussians inside $\Omega_{\text{unk}}$, and those Gaussians are barred from entering $\Omega_{\text{free}}$. This answers the problem raised by BIGS's design: its generator-driven completion has no knowledge of where the fingers went.
Baselines
The do-nothing reference is HOLD run as released, with its structure-from-motion poses. Its reported numbers on HO3D-v3 are a hand MPJPE (mean per-joint position error) of 24.2 mm and an object Chamfer distance of 0.4 cm² (HOLD). It is the natural floor because it already uses contact but neither refines poses jointly nor separates hidden space.
The strongest existing competitor is BIGS. It adds diffusion-based completion and a contact term, and it reports roughly a 40% reduction in object Chamfer distance relative to HOLD on ARCTIC, about 2.5 mm better MPJPE, and PSNR rising from about 12.8 to 24.8 in bimanual settings (BIGS). The file does not give BIGS's absolute ARCTIC Chamfer value, so we will rerun it. BIGS is the right opponent because it targets exactly the unseen-geometry problem with the most capable prior available, but it takes poses as given.
A third baseline is a pose-refining but hand-unaware pipeline. Look Gauss–style Gaussian pose refinement is run on the object with 2D hand masks and its anisotropy loss. This tests whether ordinary joint refinement with masking is enough.
All methods receive the same frames, the same initial hand fits, the same initial object poses, the same per-sequence GPU-hour budget and the same hyperparameter search. Where BIGS's prior is used, our variant uses the same diffusion model. Beating BIGS means two things. On ARCTIC and HO3D, our full method must have lower Chamfer distance on the never-observed part of the object surface, and lower object pose error, without worse visible-surface Chamfer distance or MPJPE.
The decisive ablations remove one ingredient each:
- the hand-swept free set;
- gradient routing, by letting image gradients move positions again while keeping every loss term;
- pose refinement, by freezing $T_t$.
Why not just…?
Why not just add a per-frame hand-object penetration loss, as HOLD and BIGS already do? This is the most serious threat. Summed over frames, a penetration loss is mathematically close to carving with $F^{\text{hand}}$. The difference we claim is not the term itself but where its gradients go, and whether image error can still push Gaussians into hidden space. The coupled-gradient ablation settles this, because it keeps every loss term including penetration and removes only the routing. If that ablation matches the full method on hidden-surface Chamfer distance and pose error, the routing claim is wrong. The contribution would then shrink to a pose-refinement version of HOLD, and we will report it that way.
Why not just mask the hands with a video segmenter, run structure-from-motion on the object, and inpaint? Split4D documents that tracker-dependent decomposition breaks under fast, entangled motion: 0.592 mIoU for tracking-based SA4D versus 0.893 tracker-free. HOLD itself reports structure-from-motion failures on textureless objects. Masking also discards the one signal we exploit, which is the hand's volume in hidden space. Our Look Gauss + mask baseline is exactly this alternative, with refinement added to make it stronger.
Why not just rely on diffusion completion? BIGS shows diffusion completion helps. But a generative prior has no notion of where the fingers passed. Betsu-Betsu finds that interacting surfaces fully hidden in every view get extended arbitrarily into empty space, and fused or phantom geometry inflates its Chamfer baseline to 13.89 versus 5.51. Our design keeps SDS available, but only inside $\Omega_{\text{unk}}$. The comparison of BIGS against ours-with-SDS isolates whether hand carving adds anything on top of the prior.
Why not just estimate poses first with off-the-shelf depth or odometry, then fit geometry? JOSH reports that sequential pipelines leave humans floating or sliding. Joint optimization cut its foot-floating rate from 9.0% to 2.9% and WA-MPJPE by about 46% on SLOPER4D. Its setting uses a body model against a static scene, though, so we test the hand-object case directly with the frozen-pose ablation.
Proposed Experiments
Experiments run on HO3D-v3 and ARCTIC with the official ground-truth meshes. The central metric is new. Using the ground-truth object mesh, ground-truth poses and hand meshes, we label every object surface point that is never visible, unoccluded, in any input frame, and report Chamfer distance and F-score on those points separately from visible points. No reported result was found on Lacuna for this hidden-surface Chamfer metric on either dataset, so all methods will be rerun under it.
Alongside it we report:
- object rotation and translation error per frame;
- hand MPJPE;
- penetration depth and intersection volume, following CPF;
- a needle-Gaussian statistic measuring how elongated Gaussians become, as a direct readout of the "cheating geometry" failure.
Robustness is stressed in two ways. We inject rotation errors of up to about 15° into the initial object poses, roughly the limit Look Gauss reports withstanding. We also select low-texture objects, where image gradients alone should fail.
The decisive check is a correlation within each sequence. The gain from the hand-swept set should grow with how much hidden surface the fingers actually sweep, and should be near zero for objects held only by the visible side. If the gain is uniform regardless of sweep coverage, the mechanism is not what we claim.
A secondary check runs on BEHAVE, substituting the full-body model for the hand. It asks whether body-swept carving transfers to large objects, against CARI4D's reported 38% Chamfer improvement over prior trackers there. CARI4D selects among discrete pose hypotheses rather than refining poses differentiably.
References
- EnerGS: Energy-Based Gaussian Splatting with Partial Geometric Priors
- HOLD: Category-Agnostic 3D Reconstruction of Interacting Hands and Objects from Video
- Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization
- CPF: Learning a Contact Potential Field to Model the Hand-object Interaction
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Betsu-Betsu: Multi-View Separable 3D Reconstruction of Two Interacting Objects
- Joint Optimization for 4D Human-Scene Reconstruction in the Wild
- BEHAVE: Dataset and Method for Tracking Human Object Interactions
- CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction