TL;DR
HaNDF introduces an object-conditioned generative prior for hand poses that operates directly on the geometric manifold of rotations. By learning a neural distance field that accounts for the specific object being held, it allows researchers to "project" physically impossible hand poses into realistic, contact-aware configurations for tasks like denoising, completion, and 3D reconstruction.
What Problem Does This Paper Address?
Modeling how human hands interact with objects is a cornerstone of robotics, AR/VR, and 3D computer vision. However, it is notoriously difficult. Hands are highly articulated, and the space of "valid" poses is extremely constrained by the object under interaction. A slight shift in a finger's joint angle can mean the difference between a firm grasp and a physically impossible interpenetration or a complete loss of contact.
Existing generative priors often fall into two traps:
- They are unconditional: They know what a "natural" hand looks like in a vacuum, but they don't know how a hand should adapt to a specific bottle, camera, or mug.
- They operate in Euclidean space: Most models treat joint rotations as simple vectors. But rotations live on non-linear manifolds (like spheres). Treating them as flat vectors leads to "geometric drift" and artifacts that lack physical meaning.
HaNDF (Hand Neural Distance Field) bridges this gap by creating a prior that is both geometry-aware and object-conditioned.
Methods
The core innovation of HaNDF is representing the set of plausible hand poses as the zero-level set of a neural distance field (NDF).
1. Riemannian Manifold Representation
Instead of using standard Euler angles or vectors, the authors represent the 15 local joint rotations of the MANO hand model as unit quaternions. These rotations exist on a product of 3-spheres ($S^3$). By working in this Riemannian space, the model can calculate true "geodesic" distances—the shortest path along the curved surface of the rotation space—rather than straight-line Euclidean distances that don't make sense for rotations.
2. Object Conditioning
The model doesn't just look at the hand. It takes a latent code representing the object's geometry (generated via a pretrained DeepSDF auto-decoder). This allows the distance field to "warp" based on the object. If you're holding a thin pen, the manifold of plausible poses is different than if you're holding a thick power tool.
3. Transformer Architecture
The authors use a Transformer encoder to jointly process hand-joint tokens and the object token. A special [DIST] token aggregates this information to predict a single scalar: the geodesic distance from the current hand pose to the nearest "plausible" pose for that specific object.
4. HaNDFGrad
To fix a bad pose, the authors introduce a projection operator. By taking the Riemannian gradient of the distance field, they can iteratively "push" a noisy or implausible hand pose toward the zero-level set (the manifold of valid poses).
Datasets
The model was trained and evaluated on three major large-scale hand-object interaction datasets:
- HOI4D: Egocentric RGB-D sequences featuring 800 object instances across various rigid categories.
- OakInk: A diverse repository of hand-object interactions in multi-view RGB sequences.
- DexYCB: A benchmark for capturing hand grasping of 21 YCB objects from a third-person perspective.
Key Insights and Ideas
- The Object as a Modulator: The most powerful idea here is that the object doesn't just "influence" the pose; it defines the very manifold of what is possible. By conditioning the NDF on the object, the model learns "affordances"—it understands that certain parts of an object are meant to be grasped in specific ways.
- Geometry over Latent Spaces: Unlike VAEs or Diffusion models that often operate in abstract latent spaces where distances are hard to interpret, HaNDF’s distance has a physical unit (degrees of rotation). This makes the optimization process much more stable and interpretable.
- Plug-and-Play Prior: HaNDF isn't just a standalone generator; it’s a "queryable" prior. You can take an existing, imperfect reconstruction pipeline (like EasyHOI) and use HaNDF as a post-processing step to "snap" the hand into a physically plausible configuration.
Key Results
HaNDF consistently outperformed state-of-the-art baselines across multiple tasks:
- Grasp Generation: On HOI4D, HaNDF achieved a significantly better FID (0.5466) and dNN (4.77°) compared to unconditional priors like H-NRDF, indicating higher realism and better alignment with the data distribution.
- Denoising: Even under heavy noise (1.5 rad), HaNDF recovered poses with much lower rotation errors than competitors, effectively "resolving" penetrations into objects.
- Reconstruction: When integrated into existing pipelines (AlignSDF + HaNDF), it reduced per-vertex position errors (MPVPE) and significantly decreased the Intersection Volume (I.V.), meaning fewer fingers clipping through objects.
Limitations and Future Work
While HaNDF is a major step forward, the authors note a few areas for growth:
- Object Accuracy: The model relies on reasonably accurate object geometry. If the object detection is poor, the hand prior will be conditioned on the wrong "shape," leading to poor results.
- Static vs. Dynamic: Currently, HaNDF focuses on static poses. Extending this to temporal sequences (motion priors) is a key next step.
- Bimanual Interaction: The current version is specialized for single-hand interactions. Future work aims to extend this to two-handed manipulation, which introduces even more complex constraints.
- Joint Field: The authors suggest that a "true" grasp prior should model the hand and object jointly in a single NDF, rather than conditioning one on the other.
Summary updated