H2ONet: Hand-Occlusion-and-Orientation-Aware Network for Real-Time 3D Hand Mesh Reconstruction

Hao Xu, Tianyu Wang, Xiao Tang, Chi-Wing Fu

Abstract

H2ONet is a real-time deep learning framework designed to reconstruct 3D hand meshes even when the hand is severely hidden by objects. By decoupling hand shape from global orientation and using a multi-frame fusion strategy guided by predicted occlusion levels, it achieves state-of-the-art accuracy while maintaining high-speed performance (up to 43 FPS).

Introduction

Reconstructing a 3D hand mesh from a single RGB image is a cornerstone of Augmented Reality (AR) and human-computer interaction. However, most current models stumble when the hand is "in the wild"—specifically when it's holding an object. When a hand grasps a mug or a tool, large portions of the fingers or palm disappear from view (occlusion).

Single-frame methods often "hallucinate" incorrect poses because they lack enough visual data. While some multi-frame methods exist to provide temporal context, they are often too slow for real-time use or fail to explicitly model which parts of the hand are hidden, leading to blurred or inaccurate reconstructions. H2ONet aims to bridge this gap by being both occlusion-aware and fast enough for live applications.

Methodology

The core innovation of H2ONet lies in its two-branch architecture and its occlusion-aware fusion modules.

Decoupled Learning

Instead of trying to predict the final 3D hand pose all at once, the authors split the task. One branch reconstructs the hand mesh in a "canonical" (standardized) pose, focusing purely on finger deformation. The second branch regresses the global orientation (rotation). This separation makes it much easier to fuse information across different time frames because it removes the "noise" of changing global rotations.

Occlusion-Aware Fusion

The network predicts the probability that each finger is occluded in the current and previous frames. It then uses these probabilities as weights. If the index finger is hidden in the current frame but was visible two frames ago, the network "borrows" features from that past frame to inform the current reconstruction. For global rotation, the model uses a "frame-by-frame enquiry strategy," searching backward through recent frames to find the nearest non-occluded view of the hand to help correct the orientation estimate.

The H2ONet architecture featuring a dual-branch encoder for decoupled hand mesh reconstruction and orientation regression.

Efficiency

To stay real-time, the model uses lightweight components like MobileNet and SENet blocks. For the mesh reconstruction, it employs a "SpiralConv" decoder, which is more computationally efficient than standard graph convolutions, allowing the model to maintain high frame rates.

Experimental Results

The authors evaluated H2ONet on two challenging benchmarks focused on hand-object interaction: Dex-YCB, containing over 582,000 frames of subjects grasping 20 different objects, and HO3D-v2, a widely used dataset for 3D hand-object pose estimation.

H2ONet outperformed existing state-of-the-art methods (like HandOccNet and MobRecon) across nearly all metrics. On the Dex-YCB dataset, it achieved a Joint Position Error (J-PE) of 13.7mm, compared to 14.0mm for the previous best method. Qualitative results show H2ONet successfully reconstructing hands that are almost entirely "buried" inside objects, where other models produce distorted or physically impossible meshes.

Quantitative comparison of H2ONet against state-of-the-art methods on Dex-YCB and HO3D-v2 datasets.

In terms of speed, the model runs at 35 FPS with multi-frame input and 43 FPS with single-frame input on an NVIDIA RTX 2080Ti. This performance makes it viable for real-time AR/VR applications.

Key Insights

  • Rotation is the Enemy of Fusion: Fusing features across frames is difficult because the hand's orientation changes. Transforming everything into a "canonical pose" first allows for more effective feature fusion.
  • Explicit vs. Implicit Attention: Rather than relying on generic attention mechanisms, H2ONet explicitly predicts occlusion. This provides a clear, interpretable guide for the network on which temporal data is trustworthy.
  • Temporal Redundancy: The video stream acts as a memory bank. While a finger might be occluded now, it likely wasn't occluded a few milliseconds ago, allowing the network to fill in visual gaps.

Limitations and Future Work

Despite its strengths, the authors noted a few areas for improvement. The fusion logic assumes the hand's transformation between nearby frames is mostly rigid, which might lead to errors during extremely fast, non-rigid finger movements. Additionally, the orientation branch can overfit if the training dataset lacks diverse hand rotations. Future work may involve exploring more complex temporal modeling, such as Transformers, while maintaining real-time constraints.

Summary updated

Abstract

Real-time 3D hand mesh reconstruction is challenging, especially when the hand is holding some object. Beyond the previous methods, we design H2ONet to fully exploit non-occluded information from multiple frames to boost the reconstruction quality. First, we decouple hand mesh reconstruction into two branches, one to exploit finger-level nonoccluded information and the other to exploit global hand orientation, with lightweight structures to promote real-time inference. Second, we propose finger-level occlusionaware feature fusion, leveraging predicted finger-level occlusion information as guidance to fuse finger-level information across time frames. Further, we design hand-level occlusion-aware feature fusion to fetch non-occluded information from nearby time frames. We conduct experiments on the Dex-YCB and HO3D-v2 datasets with challenging hand-object occlusion cases, manifesting that H2ONet is able to run in real-time and achieves state-of-the-art performance on both the hand mesh and pose precision. The code will be released on GitHub.
BibTeX
@inproceedings{xu2023h,
  author = {Hao Xu and Tianyu Wang and Xiao Tang and Chi-Wing Fu},
  title = {{H2ONet: Hand-Occlusion-and-Orientation-Aware Network for Real-Time 3D Hand Mesh Reconstruction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01635},
}
Figure