THOR-Net: End-to-end Graformer-based Realistic Two Hands and Object Reconstruction with Self-supervision

Ahmed Tawfik Aboukhadra, Jameel Malik, Ahmed Elhayek, Nadia Robertini, Didier Stricker

TL;DR

THOR-Net is an end-to-end framework that reconstructs 3D poses and realistic, textured meshes for two hands interacting with an object from a single RGB image. By combining Graph Convolutional Networks (GCNs) with Transformers and a self-supervised photometric loss, it achieves state-of-the-art results in hand-object reconstruction, even under heavy occlusion.

What Problem Does This Paper Address?

Reconstructing human-object interactions in 3D is a cornerstone of immersive Augmented and Virtual Reality (AR/VR). However, doing this with two hands and an object from a single 2D image is notoriously difficult.

The challenges are three-fold:

  1. Occlusion: Hands often hide parts of the object, each other, or even themselves (self-occlusion).
  2. Ambiguity: Mapping 2D pixels to 3D space is mathematically "under-determined"—there are many 3D configurations that could result in the same 2D image.
  3. Realism: Most models estimate "naked" meshes. Creating personalized, realistic textures for these meshes usually requires complex parametric models that don't always generalize well.

Methods

THOR-Net (Two Hands and Object Reconstruction Network) uses a two-stage pipeline:

1. Feature Extraction

The system starts with a Keypoint RCNN backbone. This stage extracts 2D poses, heatmaps, and Region-of-Interest (RoI) features from the input image. Crucially, the authors found that using heatmaps as node features in a graph is more accurate than using raw 2D coordinates.

2. The Reconstruction Stage

This stage splits into two specialized branches using GraFormers—a hybrid architecture that combines the local structural awareness of GCNs with the global context-gathering power of Transformers.

  • Pose Branch: Lifts 2D information into 3D poses for both hands and the object.
  • Shape Branch: Uses a novel Coarse-to-fine GraFormer. It starts with a sparse graph (the pose) and gradually increases the number of vertices (unpooling) while refining the features to produce a high-resolution 3D mesh.

3. Self-Supervised Texture

To achieve realism, the authors implement a self-supervised photometric loss. Instead of relying on a pre-defined texture model, the network projects the estimated 3D mesh back onto the 2D image and "learns" the color (RGB) for each vertex directly from the original pixels. This allows the model to capture personalized details like skin tone or text on an object.

Datasets

The model was trained and evaluated on three primary datasets:

  • HO-3D (v2 & v3): A standard benchmark for a single hand interacting with an object.
  • H2O: A challenging dataset featuring two hands and an object captured from an egocentric (first-person) perspective.
  • YCB: Used to provide the base 3D models for the objects involved in the interactions.

Key Insights and Ideas

  • The GraFormer Advantage: By using GraFormers, the model can correlate visible parts of the hand with occluded parts. The Transformer layers allow the network to "attend" to the whole hand-object relationship, solving depth ambiguities that trip up standard GCNs.
  • Coarse-to-Fine is Key: Generating a mesh all at once is prone to errors. Gradually increasing the vertex count (from 29 nodes to over 700 for hands) allows the network to learn global structure before fine-tuning local geometry.
  • Direct Texture Regression: Moving away from parametric texture models (like HTML) to direct regression via photometric loss makes the system more flexible. It can learn textures from any large-scale RGB dataset without needing specialized 3D texture scans.

Key Results

THOR-Net achieved state-of-the-art (SOTA) performance across multiple metrics:

  • Hand Shape: On the HO-3D (v3) dataset, it achieved a mesh error of 10.0mm, outperforming the previous SOTA, ArtiBoost (10.8mm).
  • Two-Hand Pose: On the H2O dataset, it improved pose estimation by 5mm for the left hand and 1mm for the right hand compared to existing methods.
  • Qualitative Realism: The photometric loss successfully captured fine details, such as the blue color and text on a book or the specific skin tones of the users, even under varying lighting.

Limitations and Future Work

While THOR-Net is a significant step forward, the authors note a few areas for improvement:

  • Generalization: The model currently struggles to generalize to completely unseen object categories that differ significantly from the training set's topologies.
  • Temporal Jitter: The current version processes images frame-by-frame. The authors suggest that incorporating temporal constraints (looking at video sequences rather than still images) could make the reconstructions smoother and more stable.
  • Texture Artifacts: Some artifacts still appear at the edges of objects where the projection mapping becomes difficult.

Summary updated

Abstract

Realistic reconstruction of two hands interacting with objects is a new and challenging problem that is essential for building personalized Virtual and Augmented Reality environments. Graph Convolutional networks (GCNs) allow for the preservation of the topologies of hands poses and shapes by modeling them as a graph. In this work, we propose the THOR-Net which combines the power of GCNs, Transformer, and self-supervision to realistically reconstruct two hands and an object from a single RGB image. Our network comprises two stages; namely the features extraction stage and the reconstruction stage. In the features extraction stage, a Keypoint RCNN is used to extract 2D poses, features maps, heatmaps, and bounding boxes from a monocular RGB image. Thereafter, this 2D information is modeled as two graphs and passed to the two branches of the reconstruction stage. The shape reconstruction branch estimates meshes of two hands and an object using our novel coarse-to-fine GraFormer shape network. The 3D poses of the hands and objects are reconstructed by the other branch using a GraFormer network. Finally, a self-supervised photometric loss is used to directly regress the realistic textured of each vertex in the hands' meshes. Our approach achieves State-of-the-art results in Hand shape estimation on the HO-3D dataset (10.0mm) exceeding ArtiBoost (10.8mm). It also surpasses other methods in hand pose estimation on the challenging two hands and object (H2O) dataset by 5mm on the left-hand pose and 1 mm on the right-hand pose.
BibTeX
@inproceedings{aboukhadra2023thor,
  author = {Ahmed Tawfik Aboukhadra and Jameel Malik and Ahmed Elhayek and Nadia Robertini and Didier Stricker},
  title = {{THOR-Net: End-to-end Graformer-based Realistic Two Hands and Object Reconstruction with Self-supervision}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00106},
}

Loading related content...

Figure