TL;DR
THOR-Net is an end-to-end framework that reconstructs 3D poses and realistic, textured meshes for two hands interacting with an object from a single RGB image. By combining Graph Convolutional Networks (GCNs) with Transformers and a self-supervised photometric loss, it achieves state-of-the-art results in hand-object reconstruction, even under heavy occlusion.
What Problem Does This Paper Address?
Reconstructing human-object interactions in 3D is a cornerstone of immersive Augmented and Virtual Reality (AR/VR). However, doing this with two hands and an object from a single 2D image is notoriously difficult.
The challenges are three-fold:
- Occlusion: Hands often hide parts of the object, each other, or even themselves (self-occlusion).
- Ambiguity: Mapping 2D pixels to 3D space is mathematically "under-determined"—there are many 3D configurations that could result in the same 2D image.
- Realism: Most models estimate "naked" meshes. Creating personalized, realistic textures for these meshes usually requires complex parametric models that don't always generalize well.
Methods
THOR-Net (Two Hands and Object Reconstruction Network) uses a two-stage pipeline:
1. Feature Extraction
The system starts with a Keypoint RCNN backbone. This stage extracts 2D poses, heatmaps, and Region-of-Interest (RoI) features from the input image. Crucially, the authors found that using heatmaps as node features in a graph is more accurate than using raw 2D coordinates.
2. The Reconstruction Stage
This stage splits into two specialized branches using GraFormers—a hybrid architecture that combines the local structural awareness of GCNs with the global context-gathering power of Transformers.
- Pose Branch: Lifts 2D information into 3D poses for both hands and the object.
- Shape Branch: Uses a novel Coarse-to-fine GraFormer. It starts with a sparse graph (the pose) and gradually increases the number of vertices (unpooling) while refining the features to produce a high-resolution 3D mesh.
3. Self-Supervised Texture
To achieve realism, the authors implement a self-supervised photometric loss. Instead of relying on a pre-defined texture model, the network projects the estimated 3D mesh back onto the 2D image and "learns" the color (RGB) for each vertex directly from the original pixels. This allows the model to capture personalized details like skin tone or text on an object.
Datasets
The model was trained and evaluated on three primary datasets:
- HO-3D (v2 & v3): A standard benchmark for a single hand interacting with an object.
- H2O: A challenging dataset featuring two hands and an object captured from an egocentric (first-person) perspective.
- YCB: Used to provide the base 3D models for the objects involved in the interactions.
Key Insights and Ideas
- The GraFormer Advantage: By using GraFormers, the model can correlate visible parts of the hand with occluded parts. The Transformer layers allow the network to "attend" to the whole hand-object relationship, solving depth ambiguities that trip up standard GCNs.
- Coarse-to-Fine is Key: Generating a mesh all at once is prone to errors. Gradually increasing the vertex count (from 29 nodes to over 700 for hands) allows the network to learn global structure before fine-tuning local geometry.
- Direct Texture Regression: Moving away from parametric texture models (like HTML) to direct regression via photometric loss makes the system more flexible. It can learn textures from any large-scale RGB dataset without needing specialized 3D texture scans.
Key Results
THOR-Net achieved state-of-the-art (SOTA) performance across multiple metrics:
- Hand Shape: On the HO-3D (v3) dataset, it achieved a mesh error of 10.0mm, outperforming the previous SOTA, ArtiBoost (10.8mm).
- Two-Hand Pose: On the H2O dataset, it improved pose estimation by 5mm for the left hand and 1mm for the right hand compared to existing methods.
- Qualitative Realism: The photometric loss successfully captured fine details, such as the blue color and text on a book or the specific skin tones of the users, even under varying lighting.
Limitations and Future Work
While THOR-Net is a significant step forward, the authors note a few areas for improvement:
- Generalization: The model currently struggles to generalize to completely unseen object categories that differ significantly from the training set's topologies.
- Temporal Jitter: The current version processes images frame-by-frame. The authors suggest that incorporating temporal constraints (looking at video sequences rather than still images) could make the reconstructions smoother and more stable.
- Texture Artifacts: Some artifacts still appear at the edges of objects where the projection mapping becomes difficult.
Summary updated