Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images

Tze Ho Elden Tse, Franziska Mueller, Zhengyang Shen, Danhang Tang, Thabo Beeler, Mingsong Dou, Yinda Zhang, Sasa Petrovic, Hyung Jin Chang, Jonathan Taylor, Bardia Doosti

Introduction

Reconstructing 3D hand meshes from an egocentric (first-person) perspective is a critical challenge for augmented and virtual reality (AR/VR). This task is particularly difficult when two hands interact, as they frequently occlude one another and share similar visual characteristics, leading to identification errors. Furthermore, most existing models neglect the forearm, which is essential for a realistic sense of presence. Spectral Graphormer addresses these issues by introducing a novel framework that combines Transformer architectures with spectral graph theory to reconstruct high-fidelity 3D meshes of both hands and forearms from multi-view RGB images.

Methodology

The Spectral Graphormer architecture introduces three primary innovations to improve efficiency and accuracy in hand reconstruction:

  1. Soft-Attention Fusion: Rather than computationally expensive image stacking, this module employs an attention mechanism to selectively fuse the most relevant features from multiple camera views.
  2. Spectral Graph-based Transformer: Utilizing spectral clustering, the model segments the hand mesh into geometrically significant regions (e.g., fingers vs. palm). These regions serve as "tokens" for the Transformer, allowing it to process the hand as a structured graph rather than an unorganized point cloud. The Graph Laplacian is further used for positional encoding, helping the model understand 3D spatial relationships.
  3. Spectral Graph Decoder: A coarse-to-fine decoder uses spectral filtering to upsample predictions into a smooth, physically plausible high-resolution mesh. A final collision refinement step ensures that reconstructed hands do not exhibit self-penetration.
The Spectral Graphormer architecture, featuring Soft-Attention Fusion, a Transformer Encoder with vertex queries, and a Spectral Graph Decoder.

Datasets

To overcome the scarcity of high-quality egocentric data, the researchers developed two new datasets:

  • Synthetic Dataset: 1 million data points with diverse lighting and complex interactions rendered from egocentric views.
  • Real-World Studio Dataset: Captured with an 18-camera rig to provide a high-precision ground truth for real-world validation.
  • FreiHAND: A standard benchmark used to evaluate the model against existing single-view reconstruction methods.

Results and Discussion

Spectral Graphormer achieves state-of-the-art performance while significantly reducing computational overhead. On the synthetic dataset, the model reached a Mean-Per-Vertex-Error (MPVE) of 1.38mm, compared to 7.09mm for the METRO baseline. Remarkably, the model size was reduced by over 50% (from 116M to 58M parameters) without compromising accuracy.

Qualitative results showing the input egocentric images and the corresponding high-fidelity 3D reconstructions from multiple viewpoints.

Qualitative analysis demonstrates that the model is robust against extreme self-occlusions and varied lighting conditions. While the system handles most interactions gracefully, extremely complex entanglements (like tightly interlaced fingers) remain a challenge for future refinement.

Conclusion

By leveraging spectral graph theory within a Transformer framework, Spectral Graphormer provides a highly efficient and accurate solution for egocentric two-hand reconstruction. The reduction in model size and the inclusion of forearms represent significant steps toward deploying high-fidelity hand tracking on resource-constrained AR/VR hardware.

Summary updated

Abstract

We propose a novel transformer-based framework that reconstructs two high fidelity hands from multi-view RGB images. Unlike existing hand pose estimation methods, where one typically trains a deep network to regress hand model parameters from single RGB image, we consider a more challenging problem setting where we directly regress the absolute root poses of two-hands with extended forearm at high resolution from egocentric view. As existing datasets are either infeasible for egocentric viewpoints or lack background variations, we create a large-scale synthetic dataset with diverse scenarios and collect a real dataset from multi-calibrated camera setup to verify our proposed multi-view image feature fusion strategy. To make the reconstruction physically plausible, we propose two strategies: (i) a coarse-to-fine spectral graph convolution decoder to smoothen the meshes during upsampling and (ii) an optimisation-based refinement stage at inference to prevent self-penetrations. Through extensive quantitative and qualitative evaluations, we show that our framework is able to produce realistic two-hand reconstructions and demonstrate the generalisation of synthetic-trained models to real data, as well as real-time AR/VR applications.
BibTeX
@inproceedings{tse2023spectral,
  author = {Tze Ho Elden Tse and Franziska Mueller and Zhengyang Shen and Danhang Tang and Thabo Beeler and Mingsong Dou and Yinda Zhang and Sasa Petrovic and Hyung Jin Chang and Jonathan Taylor and Bardia Doosti},
  title = {{Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01348},
}
Figure