Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images

Tze Ho Elden Tse, Franziska Mueller, Zhengyang Shen, Danhang Tang, Thabo Beeler, Mingsong Dou, Yinda Zhang, Sasa Petrovic, Hyung Jin Chang, Jonathan Taylor, Bardia Doosti

Abstract

We propose a novel transformer-based framework that reconstructs two high-fidelity hands from multi-view RGB images. Unlike existing hand pose estimation methods, which typically regress hand model parameters from a single RGB image, we address the more challenging problem of directly regressing the absolute root poses of two hands with extended forearms at high resolution from an egocentric view. To support this, we created a large-scale synthetic dataset and collected a real dataset from a multi-calibrated camera setup. Our framework employs a coarse-to-fine spectral graph convolution decoder to smoothen meshes and an optimization-based refinement stage to prevent self-penetrations. Quantitative and qualitative evaluations demonstrate that our framework produces realistic two-hand reconstructions and generalizes well to real-world AR/VR applications.

Introduction

Reconstructing 3D hands from a first-person (egocentric) perspective is a cornerstone of AR/VR, yet it remains incredibly difficult. Most existing methods struggle with three main issues:

  1. Two-Hand Interaction: When hands interact, they occlude each other and look very similar, leading to "self-similarity" errors where the model confuses left and right parts.
  2. Absolute Positioning: Many models predict hand poses relative to a root joint (like the wrist), but immersive VR requires knowing exactly where the hands are in 3D space relative to the cameras.
  3. Lack of Data: Existing datasets often lack the background variety, high-resolution meshes, or specific egocentric viewpoints needed to train robust models.

Furthermore, most models ignore the forearms, which are essential for a realistic sense of "presence" in virtual environments.

Methodology

The Spectral Graphormer framework introduces several novel components to tackle these challenges:

  • Soft-Attention Multi-View Fusion: Instead of simply stacking images, the model uses an attention mechanism to identify which camera view provides the best information for specific regions of the hand mesh. This makes the feature extraction "region-specific."
  • Spectral Graphormer: The core architecture is a Transformer that incorporates Spectral Graph Theory. It uses the eigenvectors of the graph Laplacian (the mathematical "DNA" of the hand's shape) to segment the mesh and provide positional encodings. This helps the Transformer understand the hand's structure without needing a rigid coordinate system.
  • Spectral Graph Decoder: To move from a low-resolution "coarse" mesh to a high-resolution "fine" mesh, the authors use a hierarchical decoder with spectral filtering. This prevents the "jittery" or distorted surfaces often seen when using standard fully-connected layers for upsampling.
  • Inference-Time Refinement: To ensure physical plausibility, an optimization step during inference detects and resolves vertex penetrations (e.g., a finger clipping through the palm).
Overview of the Spectral Graphormer architecture, featuring soft-attention fusion and a spectral graph decoder.

Datasets

Because existing data was insufficient, the team produced two new resources:

  • Synthetic Dataset: A massive collection of 1 million data points featuring high-fidelity 3D hand models with realistic textures, rendered in diverse egocentric scenarios with varying lighting and backgrounds.
  • Real Dataset: 61,000 frames captured in a specialized 18-camera studio. They used a NeRF-based (Neural Radiance Fields) pipeline to create high-quality ground truth registrations for these real-world images.

Key Results and Discussion

Spectral Graphormer achieves state-of-the-art accuracy while maintaining high efficiency. On the synthetic dataset, the model achieved a Mean-Per-Vertex-Error (MPVE) of 1.38mm, a massive improvement over the METRO baseline's 7.09mm.

Qualitative results showing high-fidelity two-hand reconstruction from egocentric input images across multiple views.

Key insights from the evaluation include:

  • Efficiency via Tokenization: By using region-specific tokens rather than global image features, the team reduced the model size to 58.3M parameters, compared to METRO’s 116.1M, while significantly improving accuracy.
  • Spectral Advantages: Using the spectral domain allows the model to treat the hand mesh as a signal on a graph, learning features that are naturally smooth across the hand's surface.
  • Physical Plausibility: The refinement stage reduced maximum penetration depth from 5.3mm to just 0.16mm, making the reconstructions look much more realistic.

Conclusion

Spectral Graphormer provides a robust solution for high-fidelity two-hand reconstruction in egocentric views. While the model handles complex poses well, the authors note that highly complex self-penetrations can still be challenging. Future work will focus on incorporating temporal information to further improve stability and extending the framework to handle hand-object interactions.

Summary updated

Abstract

We propose a novel transformer-based framework that reconstructs two high fidelity hands from multi-view RGB images. Unlike existing hand pose estimation methods, where one typically trains a deep network to regress hand model parameters from single RGB image, we consider a more challenging problem setting where we directly regress the absolute root poses of two-hands with extended forearm at high resolution from egocentric view. As existing datasets are either infeasible for egocentric viewpoints or lack background variations, we create a large-scale synthetic dataset with diverse scenarios and collect a real dataset from multi-calibrated camera setup to verify our proposed multi-view image feature fusion strategy. To make the reconstruction physically plausible, we propose two strategies: (i) a coarse-to-fine spectral graph convolution decoder to smoothen the meshes during upsampling and (ii) an optimisation-based refinement stage at inference to prevent self-penetrations. Through extensive quantitative and qualitative evaluations, we show that our framework is able to produce realistic two-hand reconstructions and demonstrate the generalisation of synthetic-trained models to real data, as well as real-time AR/VR applications.
BibTeX
@inproceedings{tse2023spectral,
  author = {Tze Ho Elden Tse and Franziska Mueller and Zhengyang Shen and Danhang Tang and Thabo Beeler and Mingsong Dou and Yinda Zhang and Sasa Petrovic and Hyung Jin Chang and Jonathan Taylor and Bardia Doosti},
  title = {{Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01348},
}
Figure