MLPHand: Real Time Multi-View 3D Hand Mesh Reconstruction via MLP Modeling

Jian Yang, Jiakun Li, Guoming Li, Zhen Shen, Huai-Yu Wu, Zhaoxin Fan, Heng HuangarXiv 2024 · DOI · arXiv

Abstract

MLPHand is the first real-time system for reconstructing 3D hand meshes from multiple camera views. By replacing heavy, transformer-based architectures with efficient Multi-Layer Perceptrons (MLPs) and a novel "per-bone" reconstruction strategy, it achieves a 90% reduction in computational cost and a 75% reduction in parameters while matching the accuracy of current state-of-the-art methods.

Introduction

Accurate 3D hand reconstruction is a "holy grail" for immersive Virtual Reality (VR) and Human-Computer Interaction (HCI). While monocular (single-camera) methods are fast, they suffer from depth ambiguity and self-occlusion—fingers often hide behind the palm or each other. Multi-view systems solve this by looking at the hand from different angles, but they traditionally come with a massive "computational tax." Existing state-of-the-art models like POEM rely on complex cross-view feature interactions (often using Transformers) that are too slow for real-time applications. MLPHand addresses this by building a multi-view system that is both highly accurate and fast enough for real-time use.

Methodology

The researchers propose a two-pronged architecture called MLPHand, which moves away from heavy attention mechanisms in favor of streamlined MLP modeling.

Architecture of the MLPHand pipeline, including the Multi-View Hand Skeleton Estimator, the Multi-view Geometry Feature Fusion (MGFP) module, and the Skeleton2Mesh model.

Skeleton2Mesh Model

This is the core geometric engine. Instead of treating the hand as one complex blob, it uses a Per-Bone reconstruction strategy, decomposing the hand into 20 bone-wise components. Crucially, it employs Tri-Axis modeling, using three parallel, lightweight MLP networks to independently regress the $x, y,$ and $z$ offsets of vertices relative to bone midpoints. This "coordinate-separable" approach is significantly faster than standard 3D regression.

Multi-view Geometry Feature Fusion (MGFP)

A skeleton alone doesn't capture fine details like skin deformation. To fix this, the MGFP module "infuses" visual features from the original images back into the mesh. It uses "Zero FC" layers (fully-connected layers initialized with zeros). This allows the model to start with a stable, skeleton-based mesh and gradually learn to incorporate visual refinements without destabilizing the training process.

Training Strategy

The team utilized a two-stage training strategy: first training the Skeleton2Mesh module on ground-truth skeletons to learn a strong geometric prior, then training the entire pipeline (including the image-based feature fusion) to refine the results.

Experiments and Results

The model was evaluated on three widely recognized multi-view benchmarks: DexYCB-MV, HO3DV3-MV, and OakInk-MV.

MLPHand sets a new bar for efficiency in the field:

  • Speed: It operates at 71 FPS on an NVIDIA 3090 GPU (using a ResNet18 backbone), making it roughly 3.7x faster than the previous state-of-the-art (POEM).
  • Efficiency: It reduces computational complexity (Multi-Adds) by 90% and cuts the parameter count by 75%.
  • Accuracy: Despite the massive speedup, it maintains comparable accuracy. For example, on the OakInk-MV dataset, it achieved a Mean Per Vertex Position Error (MPVPE) of 6.17mm, slightly outperforming POEM's 6.2mm.
Quantitative comparison of MLPHand against state-of-the-art methods across three multi-view datasets.

Discussion

The paper demonstrates that for structured geometric tasks like hand reconstruction, heavy machinery like Transformers is not strictly necessary. Simple MLPs can perform just as well if the problem is decomposed by bone and axis. The coordinate separability drastically reduces the search space and speeds up inference. However, the authors found that a Global Spatial Descriptor (GSD) is essential to maintain alignment of local bone meshes. Additionally, borrowing a concept from models like ControlNet, the use of "Zero FC" layers ensures that visual feature infusion acts as a helpful nudge rather than a source of instability.

Conclusion

MLPHand achieves real-time multi-view 3D hand mesh reconstruction with high efficiency and accuracy. While performance is tied to the quality of initial 3D skeleton estimation, the MLP-based architecture provides a robust foundation for future work, such as handling complex two-hand interactions.

Summary updated

Abstract

Multi-view hand mesh reconstruction is a critical task for applications in virtual reality and human-computer interaction, but it remains a formidable challenge. Although existing multi-view hand reconstruction methods achieve remarkable accuracy, they typically come with an intensive computational burden that hinders real-time inference. To this end, we propose MLPHand, a novel method designed for real-time multi-view single hand reconstruction. MLP Hand consists of two primary modules: (1) a lightweight MLP-based Skeleton2Mesh model that efficiently recovers hand meshes from hand skeletons, and (2) a multi-view geometry feature fusion prediction module that enhances the Skeleton2Mesh model with detailed geometric information from multiple views. Experiments on three widely used datasets demonstrate that MLPHand can reduce computational complexity by 90% while achieving comparable reconstruction accuracy to existing state-of-the-art baselines.
BibTeX
@misc{yang2024mlphand,
  author = {Jian Yang and Jiakun Li and Guoming Li and Zhen Shen and Huai-Yu Wu and Zhaoxin Fan and Heng Huang},
  title = {{MLPHand: Real Time Multi-View 3D Hand Mesh Reconstruction via MLP Modeling}},
  year = {2024},
  eprint = {2406.16137},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.16137},
}
Figure