THOR-Net: End-to-end Graformer-based Realistic Two Hands and Object Reconstruction with Self-supervision

Ahmed Tawfik Aboukhadra, Jameel Malik, Ahmed Elhayek, Nadia Robertini, Didier Stricker

Abstract

Realistic reconstruction of two hands interacting with objects is a new and challenging problem that is essential for building personalized Virtual and Augmented Reality environments. Graph Convolutional networks (GCNs) allow for the preservation of the topologies of hands poses and shapes by modeling them as a graph. In this work, we propose the THOR-Net which combines the power of GCNs, Transformer, and self-supervision to realistically reconstruct two hands and an object from a single RGB image. Our network comprises two stages; namely the features extraction stage and the reconstruction stage. In the features extraction stage, a Keypoint RCNN is used to extract 2D poses, features maps, heatmaps, and bounding boxes from a monocular RGB image. Thereafter, this 2D information is modeled as two graphs and passed to the two branches of the reconstruction stage. The shape reconstruction branch estimates meshes of two hands and an object using our novel coarse-to-fine GraFormer shape network. The 3D poses of the hands and objects are reconstructed by the other branch using a GraFormer network. Finally, a self-supervised photometric loss is used to directly regress the realistic textured of each vertex in the hands' meshes. Our approach achieves State-of-the-art results in Hand shape estimation on the HO-3D dataset (10.0mm) exceeding ArtiBoost (10.8mm). It also surpasses other methods in hand pose estimation on the challenging two hands and object (H2O) dataset by 5mm on the left-hand pose and 1 mm on the right-hand pose.

Introduction

Reconstructing human-object interactions in 3D is a cornerstone of immersive Augmented and Virtual Reality (AR/VR). However, doing this with two hands and an object from a single 2D image is notoriously difficult due to heavy occlusion, depth ambiguity, and the challenge of achieving visual realism. THOR-Net (Two Hands and Object Reconstruction Network) addresses these hurdles by integrating Graph Convolutional Networks (GCNs) with Transformers and a self-supervised photometric loss to produce textured, high-fidelity 3D meshes.

Methodology

THOR-Net utilizes a two-stage pipeline designed to bridge the gap between 2D image data and 3D spatial understanding.

1. Feature Extraction

The system begins with a Keypoint RCNN backbone that extracts 2D poses, heatmaps, and Region-of-Interest (RoI) features. The network uses heatmaps as node features in a graph, which provides a more robust representation than raw 2D coordinates for subsequent 3D lifting.

2. The Reconstruction Stage

This stage employs GraFormers—a hybrid architecture combining the local structural awareness of GCNs with the global context-gathering power of Transformers. This allows the model to correlate visible parts of the hand with occluded sections, resolving depth ambiguities.

  • Pose Branch: Lifts 2D information into 3D poses for both hands and the object.
  • Shape Branch: Utilizes a novel Coarse-to-fine GraFormer. It begins with a sparse graph (the pose) and gradually increases vertex density through unpooling layers, refining features to produce a high-resolution 3D mesh.
The THOR-Net architecture, detailing the feature extraction via Keypoint RCNN followed by the Graph Initialization and Coarse-to-fine Shape GraFormer modules.

3. Self-Supervised Texture Regression

To achieve realism without specialized 3D texture scans, the authors implement a self-supervised photometric loss. The network projects the estimated 3D mesh back onto the 2D image and learns the RGB color for each vertex directly from the original pixels. This captures personalized details like skin tone and object labels that parametric models often miss.

Key Results and Evaluation

THOR-Net was evaluated on standard benchmarks including HO-3D (single hand-object) and the H2O dataset (two hands-object egocentric view).

  • State-of-the-Art Performance: On the HO-3D (v3) dataset, THOR-Net achieved a mesh error of 10.0mm, outperforming ArtiBoost (10.8mm). On the H2O dataset, it improved pose estimation by 5mm for the left hand and 1mm for the right hand over existing methods.
  • Qualitative Realism: The model successfully reconstructs fine details, such as text on a book or specific user skin tones, even under varying lighting conditions.
Qualitative results showing the progression from input image to 3D pose and final textured mesh reconstruction.

Conclusion

THOR-Net demonstrates that combining the structural priors of GCNs with the global attention of Transformers enables highly accurate 3D reconstruction of complex hand-object interactions. While future work is needed to address temporal stability in video and generalization to entirely novel object topologies, THOR-Net sets a new standard for realistic, end-to-end 3D hand-object modeling.

Summary updated

Abstract

Realistic reconstruction of two hands interacting with objects is a new and challenging problem that is essential for building personalized Virtual and Augmented Reality environments. Graph Convolutional networks (GCNs) allow for the preservation of the topologies of hands poses and shapes by modeling them as a graph. In this work, we propose the THOR-Net which combines the power of GCNs, Transformer, and self-supervision to realistically reconstruct two hands and an object from a single RGB image. Our network comprises two stages; namely the features extraction stage and the reconstruction stage. In the features extraction stage, a Keypoint RCNN is used to extract 2D poses, features maps, heatmaps, and bounding boxes from a monocular RGB image. Thereafter, this 2D information is modeled as two graphs and passed to the two branches of the reconstruction stage. The shape reconstruction branch estimates meshes of two hands and an object using our novel coarse-to-fine GraFormer shape network. The 3D poses of the hands and objects are reconstructed by the other branch using a GraFormer network. Finally, a self-supervised photometric loss is used to directly regress the realistic textured of each vertex in the hands' meshes. Our approach achieves State-of-the-art results in Hand shape estimation on the HO-3D dataset (10.0mm) exceeding ArtiBoost (10.8mm). It also surpasses other methods in hand pose estimation on the challenging two hands and object (H2O) dataset by 5mm on the left-hand pose and 1 mm on the right-hand pose.
BibTeX
@inproceedings{aboukhadra2023thor,
  author = {Ahmed Tawfik Aboukhadra and Jameel Malik and Ahmed Elhayek and Nadia Robertini and Didier Stricker},
  title = {{THOR-Net: End-to-end Graformer-based Realistic Two Hands and Object Reconstruction with Self-supervision}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00106},
}
Figure