When two hands touch, a single camera sees one blob of skin. Systems that rebuild both hands in 3D from such an image produce meshes that pass through each other. The usual fix pushes the two surfaces apart until they no longer overlap. Real hands that press together do not behave this way: skin compresses, fingertips flatten, and the two surfaces meet with a small overlap if you model them as rigid shells. A system trained only to remove overlap therefore learns to leave a gap. We propose to replace the "push apart" signal with a signal that does two jobs. It pulls together the surface regions the image says are touching, and it repels only penetration deeper than a soft-tissue allowance of a few millimetres. The contact predictions come from the image, read at each step of the generation process, so the 3D model is told where touching happens rather than inferring it from geometry alone.
Concretely, we use a diffusion model, a generator that learns to turn random noise into samples through repeated denoising, as the prior over two-hand meshes. We change what steers its sampling. The current best two-hand diffusion system, Aligning Foundation Model Priors and Diffusion-Based Hand Interactions, applies a purely repulsive "collision gradient guidance" and reports a penetration volume of 0.11. Our hypothesis is that part of that low number comes from hands that hover apart. We expect compliant, image-grounded contact guidance to match its joint accuracy, to place hands correctly relative to each other at the contact points, and to remove that floating.
Core Claim
Two-hand reconstruction has been tuned against penetration metrics that reward separation. ContactOpt refines hand–object grasps with a differentiable contact model. It found that treating skin as a rigid non-penetrating shell destabilises optimisation, and that allowing up to 2 mm of interpenetration gives smoother gradients and grasps people prefer by a 6:1 ratio. No two-hand method has carried this finding over to bimanual interaction. In two-hand interaction both surfaces are soft, articulated and look alike.
Our claim is this. If the denoising sampler is guided by an energy that combines (a) attraction between vertex pairs that an image-conditioned head predicts to be in contact and (b) repulsion that switches on only beyond a compliance depth, then the sampler will produce more accurate contact interfaces and relative placement than repulsion-only guidance, with no loss in joint error. The essential ingredient is the image-predicted contact term, not the tolerance threshold. Our ablations are built to separate the two.
Relation to Prior Work
Several lines of work motivate the gap.
- Aligning Foundation Model Priors… treats interpenetrated two-hand poses as noisy samples and denoises them with repulsive guidance. It has no notion of where contact should occur. - Reconstructing Interacting Hands with Interaction Prior learns a VAE prior on the physics-filtered Two-hand 500K dataset. It admits that tangled, unseen poses still drift without test-time physical constraints.
- IHMR documents a "lazy solution": collision losses applied during training make the network pull the hands apart. It fixes this with factorised post-hoc refinement, reporting a 71.4% reduction in average penetration depth on the Inter-Close subset.
All three treat any overlap as error, which puts them in direct tension with ContactOpt.
Other work supplies the method.
- CPF models hand–object interaction as attractive and repulsive springs between surfaces. Its attractive springs are driven by contact predicted from pixels. It assumes a known rigid object mesh and solves by numerical energy minimisation.
- DiffHand denoises mesh vertices directly. Its cross-modality decoder lets vertex queries attend to image features. It handles one hand only and has no collision term.
We borrow CPF's attract/repel structure, ContactOpt's compliance allowance and DiffHand's vertex-to-image attention. We apply them to two deformable hands, inside the diffusion sampler rather than in a separate optimiser.
One paper constrains the design. Countering Multi-modal Representation Collapse warns that a fusion model can ignore one input. A contact head that sees both the current mesh and the image could predict contact purely from vertex proximity. That would make it a geometric heuristic, not image evidence, so we test for this collapse directly.
Research Program
Base model. The base model is a conditional diffusion model over both hands' MANO parameters, where MANO is the standard parametric hand mesh. It is trained on InterHand2.6M and conditioned on image features. We reimplement the sampling setup of the Aligning paper as closely as its description allows, and use the released code instead if it is available. The guidance term plugs into the sampler, so it can also be applied unchanged to that system's sampler.
Contact head. At each denoising step $t$ we decode the current estimate of both meshes. A small cross-attention head lets each vertex query image features at its projected location, and outputs a per-vertex contact probability $c_i$ and a partner correspondence on the other hand. The cross-attention head follows DiffHand's decoder and also uses the Interaction Adjacency Heatmap features of Reconstructing Interacting Hands, which keep overlapping digits apart in the feature map.
Supervision for the contact head. Labels come from ground-truth MANO fits: a vertex is "in contact" if it lies within a distance threshold of the other hand's surface. The threshold is a hyperparameter that we sweep, because the fits carry error above the 2.78 mm triangulation error.
Guidance energy. The guidance energy over the current vertex estimate $V$ is
$$E(V) = \lambda_a \sum_{i} c_i \, \lVert v_i - v_{\pi(i)} \rVert^2 \;+\; \lambda_r \sum_{j \in \mathcal{P}} \max\!\big(0,\, d_j - \delta\big)^2,$$
where the terms are as follows:
- $v_i$ is a vertex and $v_{\pi(i)}$ its predicted partner on the opposite hand.
- $\mathcal{P}$ is the set of vertices lying inside the other hand.
- $d_j$ is the penetration depth of vertex $j$.
- $\delta$ is the compliance allowance, swept around ContactOpt's 2 mm.
- $\lambda_a$ and $\lambda_r$ weight attraction and repulsion.
The sampler subtracts the gradient of $E$, scaled by a step-dependent weight, exactly where the Aligning paper subtracts its repulsive gradient.
Why this should avoid the lazy solution. The guidance acts only at sampling time, and the attraction term penalises separation at predicted contacts. Moving the hands apart therefore stops being a free way to lower the energy.
Baselines
Naive reference. We run the same diffusion model with no guidance and no contact head. This measures how much any physical steering helps. As a second reference, we report the errors of the ground-truth fits themselves on penetration and contact metrics. No reported result was found on Lacuna for the penetration volume of InterHand2.6M ground truth. This number matters because it shows whether "zero penetration" is even the right target.
Strongest existing method. The strongest competitor is the Aligning Foundation Model Priors system. It is the only two-hand method in the file that uses a diffusion prior with test-time collision handling, and it reports an MRRPE of 21.60 mm and a PenVol of 0.11 on InterHand2.6M. MRRPE is the error in the position of one hand's root relative to the other, in mm. PenVol is penetration volume.
We compare against it in two ways:
- Our reimplementation: our sampler with its repulsive guidance, on the same backbone, the same number of denoising steps, the same training data, and an equal hyperparameter search budget for the guidance weights.
- Reported numbers: we cite its published figures for reference.
What counts as beating it. The bar is all of the following at once:
- no worse MPJPE (mean per-joint position error) or MRRPE;
- significantly better contact F1 and lower gap at ground-truth contact vertices;
- PenVol no higher than the ground-truth fits' own level.
Other existing methods.
- Best reported joint error: we also report the 8.34 mm MPJPE of Reconstructing Interacting Hands as the best joint-accuracy figure in the file. It comes from a different paper, so it is a reference, not a controlled comparison.
- IHMR: we compare against IHMR as reported. The file gives its gains only as relative reductions (71.4% average penetration depth, 16.5% I-MPJPE, the joint error on interacting hands), not absolute values.
Ablations.
- Compliance only: repulsion with the $\delta$ allowance but no attraction.
- Contact from proximity: attraction where contacts come from mesh proximity alone, with no image input.
- Shuffled image features: the full model with image features shuffled across samples at test time.
Why not just…?
Why not just add a 2 mm margin to the existing repulsive guidance?
This is the most serious threat. It may recover much of the benefit on PenVol and floating. The "compliance only" ablation tests it directly. If it matches the full model on contact F1 and contact gap, the image-predicted attraction is unnecessary and we will report that.
We expect it to fall short in one specific case: contacts the image shows but the coarse sample places 5–15 mm apart. A margin cannot close that gap. We will report results separately on the subset where the unguided sample's contact gap exceeds 5 mm.
Why not just run ContactOpt-style optimisation after the existing diffusion system?
ContactOpt assumes a rigid, known object, and its contact network was trained on hand–object thermal data. It has never been shown to work with two articulated hands. Post-hoc optimisation also leaves the prior behind: the optimiser can move hands to poses the diffusion model considers implausible.
We will run this baseline anyway, using our contact head and energy with ADAM after sampling. Comparing it with in-sampler guidance tells us whether the integration matters or only the energy does.
Why not just use IHMR's factorised refinement?
IHMR is deterministic, has no generative prior, and its authors report failures when one hand is nearly invisible. That case is exactly where image-predicted contact plus a prior should help. Its refinement still only resolves collisions; it never attracts. We will evaluate both methods on the most-occluded decile of InterHand2.6M interacting frames.
Why not just train on physics-filtered data?
The Reconstructing Interacting Hands authors themselves report residual penetration on tangled poses. Filtering also removes the very compliant overlaps we argue are real.
Proposed Experiments
Data and metrics. We train and test on InterHand2.6M, with test-time generalisation on Re:InterHand and the Hand-in-the-Wild (HIC) benchmark. Accuracy metrics are MPJPE, I-MPJPE, MRRPE and PenVol. We add three contact metrics computed from the ground-truth fits:
- contact F1 at matched thresholds;
- mean gap at ground-truth contact vertices, which measures floating;
- penetration depth histogram, compared against the histogram of the ground-truth fits.
Decisive checks. Three checks settle the main claims.
- Floating in the repulsive baseline. Does the repulsion-only baseline show larger contact gaps than the ground truth while reaching lower PenVol? If not, the premise of floating is wrong and the project narrows to contact accuracy.
- Value of the image term. Does the full model beat the "compliance only" and "contact from proximity" ablations on contact F1? It must do so on the large-gap subset in particular.
- Collapse test. Does shuffling image features degrade contact predictions? If they barely change, the head has collapsed onto geometry, as the collapse paper warns. We would then add the rank-targeted regularisation it proposes and re-test.
Runtime and noise checks.
- Runtime: we report frames per second against the reported 18 FPS of the Aligning pipeline. The contact head adds one cross-attention pass per denoising step, and we will measure its cost.
- Label noise: we sweep the contact-label threshold to see whether the conclusions survive noise in the ground-truth fits.
References
- Aligning Foundation Model Priors and Diffusion-Based Hand Interactions for Occlusion-Resistant Two-Hand Reconstruction
- ContactOpt: Optimizing Contact to Improve Grasps
- Reconstructing Interacting Hands with Interaction Prior from Monocular Images
- Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements
- CPF: Learning a Contact Potential Field to Model the Hand-object Interaction
- DiffHand: End-to-End Hand Mesh Reconstruction via Diffusion Models
- Countering Multi-modal Representation Collapse through Rank-targeted Fusion