HaNDF: Object-Conditioned Geometric Neural Hand Distance Fields

Zuhao Liu, Melih Darcan, Zhengdi Yu, Riza Alp Guler, Tolga BirdalarXiv 2026 · arXiv

TL;DR

HaNDF introduces an object-conditioned generative prior for hand poses that operates directly on the geometric manifold of rotations. By learning a neural distance field that accounts for the specific object being held, it allows researchers to "project" physically impossible hand poses into realistic, contact-aware configurations for tasks like denoising, completion, and 3D reconstruction.

What Problem Does This Paper Address?

Modeling how human hands interact with objects is a cornerstone of robotics, AR/VR, and 3D computer vision. However, it is notoriously difficult. Hands are highly articulated, and the space of "valid" poses is extremely constrained by the object under interaction. A slight shift in a finger's joint angle can mean the difference between a firm grasp and a physically impossible interpenetration or a complete loss of contact.

Existing generative priors often fall into two traps:

  1. They are unconditional: They know what a "natural" hand looks like in a vacuum, but they don't know how a hand should adapt to a specific bottle, camera, or mug.
  2. They operate in Euclidean space: Most models treat joint rotations as simple vectors. But rotations live on non-linear manifolds (like spheres). Treating them as flat vectors leads to "geometric drift" and artifacts that lack physical meaning.

HaNDF (Hand Neural Distance Field) bridges this gap by creating a prior that is both geometry-aware and object-conditioned.

Figure 1: We introduce HaNDF, an object-conditioned generative hand prior. HaNDF is a geometric neural distance field representing the plausible hand configurations under interaction, and enables numerous tasks from denoising and completion to recovering hands or even objects from image data.

Methods

The core innovation of HaNDF is representing the set of plausible hand poses as the zero-level set of a neural distance field (NDF).

1. Riemannian Manifold Representation

Instead of using standard Euler angles or vectors, the authors represent the 15 local joint rotations of the MANO hand model as unit quaternions. These rotations exist on a product of 3-spheres ($S^3$). By working in this Riemannian space, the model can calculate true "geodesic" distances—the shortest path along the curved surface of the rotation space—rather than straight-line Euclidean distances that don't make sense for rotations.

2. Object Conditioning

The model doesn't just look at the hand. It takes a latent code representing the object's geometry (generated via a pretrained DeepSDF auto-decoder). This allows the distance field to "warp" based on the object. If you're holding a thin pen, the manifold of plausible poses is different than if you're holding a thick power tool.

3. Transformer Architecture

The authors use a Transformer encoder to jointly process hand-joint tokens and the object token. A special [DIST] token aggregates this information to predict a single scalar: the geodesic distance from the current hand pose to the nearest "plausible" pose for that specific object.

4. HaNDFGrad

To fix a bad pose, the authors introduce a projection operator. By taking the Riemannian gradient of the distance field, they can iteratively "push" a noisy or implausible hand pose toward the zero-level set (the manifold of valid poses).

Datasets

The model was trained and evaluated on three major large-scale hand-object interaction datasets:

  • HOI4D: Egocentric RGB-D sequences featuring 800 object instances across various rigid categories.
  • OakInk: A diverse repository of hand-object interactions in multi-view RGB sequences.
  • DexYCB: A benchmark for capturing hand grasping of 21 YCB objects from a third-person perspective.

Key Insights and Ideas

  • The Object as a Modulator: The most powerful idea here is that the object doesn't just "influence" the pose; it defines the very manifold of what is possible. By conditioning the NDF on the object, the model learns "affordances"—it understands that certain parts of an object are meant to be grasped in specific ways.
  • Geometry over Latent Spaces: Unlike VAEs or Diffusion models that often operate in abstract latent spaces where distances are hard to interpret, HaNDF’s distance has a physical unit (degrees of rotation). This makes the optimization process much more stable and interpretable.
  • Plug-and-Play Prior: HaNDF isn't just a standalone generator; it’s a "queryable" prior. You can take an existing, imperfect reconstruction pipeline (like EasyHOI) and use HaNDF as a post-processing step to "snap" the hand into a physically plausible configuration.

Key Results

HaNDF consistently outperformed state-of-the-art baselines across multiple tasks:

  • Grasp Generation: On HOI4D, HaNDF achieved a significantly better FID (0.5466) and dNN (4.77°) compared to unconditional priors like H-NRDF, indicating higher realism and better alignment with the data distribution.
  • Denoising: Even under heavy noise (1.5 rad), HaNDF recovered poses with much lower rotation errors than competitors, effectively "resolving" penetrations into objects.
  • Reconstruction: When integrated into existing pipelines (AlignSDF + HaNDF), it reduced per-vertex position errors (MPVPE) and significantly decreased the Intersection Volume (I.V.), meaning fewer fingers clipping through objects.
Qualitative denoising iterations: Starting from a noisy state (t=0), HaNDF iteratively refines the hand into a physically plausible grasp.

Limitations and Future Work

While HaNDF is a major step forward, the authors note a few areas for growth:

  • Object Accuracy: The model relies on reasonably accurate object geometry. If the object detection is poor, the hand prior will be conditioned on the wrong "shape," leading to poor results.
  • Static vs. Dynamic: Currently, HaNDF focuses on static poses. Extending this to temporal sequences (motion priors) is a key next step.
  • Bimanual Interaction: The current version is specialized for single-hand interactions. Future work aims to extend this to two-handed manipulation, which introduces even more complex constraints.
  • Joint Field: The authors suggest that a "true" grasp prior should model the hand and object jointly in a single NDF, rather than conditioning one on the other.

Summary updated

Abstract

We propose, HaNDF, object-conditional neural distance fields as data-driven priors for modeling the plausible hand-object configurations, in the product manifold of hand articulations. While recent unconditional generative priors showed success in modeling human pose, hands are versatile articulated bodies often found in interaction with other objects. This makes it crucial to develop a generative prior which can model the plausible hand poses conditioned on the object under interaction. To this end, our HaNDF proposes a geometry-aware, object-conditioned neural distance field, representing the plausible articulations in the zero-level set of a conditional field. We leverage tools from Riemannian geometry to (i) represent plausible hands in the zero level-set of an object-induced neural field, and (ii) project a given pose onto this field. Using HaNDF, we can also optimize for a plausible condition, determining the object under interaction. Our extensive evaluations demonstrate that our HaNDF provides a flexible and powerful prior for hand-object interaction, suitable for applications such as pose refinement, reconstruction under occlusion, and physically plausible manipulation synthesis.
BibTeX
@misc{liu2026handf,
  author = {Zuhao Liu and Melih Darcan and Zhengdi Yu and Riza Alp Guler and Tolga Birdal},
  title = {{HaNDF: Object-Conditioned Geometric Neural Hand Distance Fields}},
  year = {2026},
  eprint = {2611.08578},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08578},
}
Figure