Abstract
MLPHand is the first real-time system for reconstructing 3D hand meshes from multiple camera views. By replacing heavy, transformer-based architectures with efficient Multi-Layer Perceptrons (MLPs) and a novel "per-bone" reconstruction strategy, it achieves a 90% reduction in computational cost and a 75% reduction in parameters while matching the accuracy of current state-of-the-art methods.
Introduction
Accurate 3D hand reconstruction is a "holy grail" for immersive Virtual Reality (VR) and Human-Computer Interaction (HCI). While monocular (single-camera) methods are fast, they suffer from depth ambiguity and self-occlusion—fingers often hide behind the palm or each other. Multi-view systems solve this by looking at the hand from different angles, but they traditionally come with a massive "computational tax." Existing state-of-the-art models like POEM rely on complex cross-view feature interactions (often using Transformers) that are too slow for real-time applications. MLPHand addresses this by building a multi-view system that is both highly accurate and fast enough for real-time use.
Methodology
The researchers propose a two-pronged architecture called MLPHand, which moves away from heavy attention mechanisms in favor of streamlined MLP modeling.
Skeleton2Mesh Model
This is the core geometric engine. Instead of treating the hand as one complex blob, it uses a Per-Bone reconstruction strategy, decomposing the hand into 20 bone-wise components. Crucially, it employs Tri-Axis modeling, using three parallel, lightweight MLP networks to independently regress the $x, y,$ and $z$ offsets of vertices relative to bone midpoints. This "coordinate-separable" approach is significantly faster than standard 3D regression.
Multi-view Geometry Feature Fusion (MGFP)
A skeleton alone doesn't capture fine details like skin deformation. To fix this, the MGFP module "infuses" visual features from the original images back into the mesh. It uses "Zero FC" layers (fully-connected layers initialized with zeros). This allows the model to start with a stable, skeleton-based mesh and gradually learn to incorporate visual refinements without destabilizing the training process.
Training Strategy
The team utilized a two-stage training strategy: first training the Skeleton2Mesh module on ground-truth skeletons to learn a strong geometric prior, then training the entire pipeline (including the image-based feature fusion) to refine the results.
Experiments and Results
The model was evaluated on three widely recognized multi-view benchmarks: DexYCB-MV, HO3DV3-MV, and OakInk-MV.
MLPHand sets a new bar for efficiency in the field:
- Speed: It operates at 71 FPS on an NVIDIA 3090 GPU (using a ResNet18 backbone), making it roughly 3.7x faster than the previous state-of-the-art (POEM).
- Efficiency: It reduces computational complexity (Multi-Adds) by 90% and cuts the parameter count by 75%.
- Accuracy: Despite the massive speedup, it maintains comparable accuracy. For example, on the OakInk-MV dataset, it achieved a Mean Per Vertex Position Error (MPVPE) of 6.17mm, slightly outperforming POEM's 6.2mm.
Discussion
The paper demonstrates that for structured geometric tasks like hand reconstruction, heavy machinery like Transformers is not strictly necessary. Simple MLPs can perform just as well if the problem is decomposed by bone and axis. The coordinate separability drastically reduces the search space and speeds up inference. However, the authors found that a Global Spatial Descriptor (GSD) is essential to maintain alignment of local bone meshes. Additionally, borrowing a concept from models like ControlNet, the use of "Zero FC" layers ensures that visual feature infusion acts as a helpful nudge rather than a source of instability.
Conclusion
MLPHand achieves real-time multi-view 3D hand mesh reconstruction with high efficiency and accuracy. While performance is tied to the quality of initial 3D skeleton estimation, the MLP-based architecture provides a robust foundation for future work, such as handling complex two-hand interactions.
Summary updated