Abstract
We propose a novel transformer-based framework that reconstructs two high-fidelity hands from multi-view RGB images. Unlike existing hand pose estimation methods, which typically regress hand model parameters from a single RGB image, we address the more challenging problem of directly regressing the absolute root poses of two hands with extended forearms at high resolution from an egocentric view. To support this, we created a large-scale synthetic dataset and collected a real dataset from a multi-calibrated camera setup. Our framework employs a coarse-to-fine spectral graph convolution decoder to smoothen meshes and an optimization-based refinement stage to prevent self-penetrations. Quantitative and qualitative evaluations demonstrate that our framework produces realistic two-hand reconstructions and generalizes well to real-world AR/VR applications.
Introduction
Reconstructing 3D hands from a first-person (egocentric) perspective is a cornerstone of AR/VR, yet it remains incredibly difficult. Most existing methods struggle with three main issues:
- Two-Hand Interaction: When hands interact, they occlude each other and look very similar, leading to "self-similarity" errors where the model confuses left and right parts.
- Absolute Positioning: Many models predict hand poses relative to a root joint (like the wrist), but immersive VR requires knowing exactly where the hands are in 3D space relative to the cameras.
- Lack of Data: Existing datasets often lack the background variety, high-resolution meshes, or specific egocentric viewpoints needed to train robust models.
Furthermore, most models ignore the forearms, which are essential for a realistic sense of "presence" in virtual environments.
Methodology
The Spectral Graphormer framework introduces several novel components to tackle these challenges:
- Soft-Attention Multi-View Fusion: Instead of simply stacking images, the model uses an attention mechanism to identify which camera view provides the best information for specific regions of the hand mesh. This makes the feature extraction "region-specific."
- Spectral Graphormer: The core architecture is a Transformer that incorporates Spectral Graph Theory. It uses the eigenvectors of the graph Laplacian (the mathematical "DNA" of the hand's shape) to segment the mesh and provide positional encodings. This helps the Transformer understand the hand's structure without needing a rigid coordinate system.
- Spectral Graph Decoder: To move from a low-resolution "coarse" mesh to a high-resolution "fine" mesh, the authors use a hierarchical decoder with spectral filtering. This prevents the "jittery" or distorted surfaces often seen when using standard fully-connected layers for upsampling.
- Inference-Time Refinement: To ensure physical plausibility, an optimization step during inference detects and resolves vertex penetrations (e.g., a finger clipping through the palm).
Datasets
Because existing data was insufficient, the team produced two new resources:
- Synthetic Dataset: A massive collection of 1 million data points featuring high-fidelity 3D hand models with realistic textures, rendered in diverse egocentric scenarios with varying lighting and backgrounds.
- Real Dataset: 61,000 frames captured in a specialized 18-camera studio. They used a NeRF-based (Neural Radiance Fields) pipeline to create high-quality ground truth registrations for these real-world images.
Key Results and Discussion
Spectral Graphormer achieves state-of-the-art accuracy while maintaining high efficiency. On the synthetic dataset, the model achieved a Mean-Per-Vertex-Error (MPVE) of 1.38mm, a massive improvement over the METRO baseline's 7.09mm.
Key insights from the evaluation include:
- Efficiency via Tokenization: By using region-specific tokens rather than global image features, the team reduced the model size to 58.3M parameters, compared to METRO’s 116.1M, while significantly improving accuracy.
- Spectral Advantages: Using the spectral domain allows the model to treat the hand mesh as a signal on a graph, learning features that are naturally smooth across the hand's surface.
- Physical Plausibility: The refinement stage reduced maximum penetration depth from 5.3mm to just 0.16mm, making the reconstructions look much more realistic.
Conclusion
Spectral Graphormer provides a robust solution for high-fidelity two-hand reconstruction in egocentric views. While the model handles complex poses well, the authors note that highly complex self-penetrations can still be challenging. Future work will focus on incorporating temporal information to further improve stability and extending the framework to handle hand-object interactions.
Summary updated