Abstract
Estimating the poses of both a hand and an object has become an important area of research due to the growing need for advanced vision computing. The primary challenge involves understanding and reconstructing how hands and objects interact, such as contact and physical plausibility. Existing approaches often adopt a graph neural network to incorporate spatial information of hand and object meshes. However, these approaches have not fully exploited the potential of graphs without modification of edges within and between hand- and object-graphs. We propose a graph-based refinement method that incorporates an interaction-aware graph-attention mechanism to account for hand-object interactions. Using edges, we establish connections among closely correlated nodes, both within individual graphs and across different graphs. Experiments demonstrate the effectiveness of our proposed method with notable improvements in the realm of physical plausibility.
Introduction
Reconstructing the interaction between a hand and an object from a single 2D image is a fundamental task for virtual and augmented reality (VR/AR) and robotics. A significant hurdle in current research is achieving physical plausibility. Many existing models process hands and objects in separate neural branches, often resulting in 3D reconstructions where the hand penetrates the object's surface or appears to float without making contact. While Graph Neural Networks (GNNs) are commonly used to model mesh structures, they typically rely on static graph topologies that do not adapt to the specific interaction dynamics of a scene. This work introduces a method to make these reconstructions "interaction-aware" at the mesh level.
Methodology
The proposed pipeline consists of three distinct stages designed to refine the spatial relationship between the hand and the object.
1. Initial Mesh Estimation
The system utilizes a ResNet-18 backbone to extract features. An initial hand mesh is generated using the MANO model, while the object mesh is estimated via AtlasNet. These initial estimations provide the starting point for the graph-based refinement.
2. Graph Initialization
The core innovation lies in the dynamic definition of edges to link vertices (nodes) within and between the hand and object meshes:
- Common Relation Edges ($E_c$): These represent physical proximity, including standard mesh faces for intra-class connectivity and edges connecting the nearest points between the hand and object for inter-class connectivity.
- Attention-Guided Edges ($E_a$): These are "soft" edges determined by self-attention and cross-attention mechanisms. Edges are formed between nodes that share high feature correlation in latent space, even if they are spatially distant, allowing for global context integration.
3. Graph Refinement
The initial meshes are processed through four Graph Convolution (GC) blocks. These blocks leverage the established edges to calculate vertex displacements ($\Delta V$), fine-tuning the position of every mesh point to ensure image alignment and physical boundary respect.
Experiments and Results
The model was evaluated on two primary benchmarks: ObMan, a large-scale synthetic dataset with physically simulated ground truth, and DexYCB, a real-world dataset of grasping interactions.
Quantitative Performance
The proposed method significantly enhances physical realism:
- Reduced Penetration: On the ObMan dataset, the model achieved an Intersection Volume (I.V.) of 3.5 $cm^3$, outperforming previous baselines such as Hasson et al. ($12.2 cm^3$) and Tse et al. ($9.3 cm^3$).
- Refinement Impact: The refinement stage alone reduced the Maximum Penetration (Max. pen.) from 8.83mm to 6.90mm on ObMan.
Qualitatively, the results demonstrate tighter, more realistic grasps where the hand mesh effectively wraps around the object's geometry rather than clipping through it.
Discussion and Conclusion
A key insight of this research is that inter-class connectivity is essential for learning boundary constraints. By forcing message passing across hand-to-object edges, the network becomes aware of the physical limits of the interaction. To maintain computational efficiency, the authors employ a sparsity strategy, connecting only designated hand nodes likely to be in contact to their nearest object neighbors.
While the method improves physical plausibility, a trade-off was observed: the Hand Joint Error (H.E.) slightly increased. This is attributed to the refinement stage moving vertices to satisfy physical constraints, which can slightly deviate from the rigid skeletal structure of the initial MANO model. Future work will focus on better integrating these physical constraints with skeletal models and improving performance on real-world datasets where ground-truth data may be noisy.
Summary updated