TL;DR
FAH (Fast and Accurate Hand Estimation) is a new family of real-time models that achieves state-of-the-art accuracy in 3D hand mesh recovery. By predicting 3D points directly and then fitting a mesh—rather than regressing rig parameters—it achieves superior pixel alignment and runs at up to 125 FPS on consumer hardware.
What Problem Does This Paper Address?
Most current monocular hand mesh estimators follow a "parameter regression" approach: they predict the internal parameters of a hand rig (like the widely used MANO model) and then project that rig to get 2D keypoints.
The authors identify a fundamental flaw here: rig constraints and linear blend skinning often prevent the resulting mesh from perfectly aligning with the actual pixels in the image. Furthermore, many existing methods struggle with temporal stability (jitter in video), lack world-space coordinates (they only provide camera-relative positions), and fail to estimate hand-object contact accurately without knowing the object's geometry.
Methods
The core philosophy of FAH is "point prediction then mesh fitting." Instead of asking the neural network to guess abstract rig parameters, the authors train it to predict 2D and 3D joints and surface points directly.
Key components include:
- Architecture: Built on the hand branch of SAM 3D Body, using a Vision Transformer (ViT) backbone. The authors provide a range of model sizes (Small, Base, Large, and Huge) to trade off speed and accuracy.
- Decoder-free Dense Head: A heatmap-based head predicts 21 joints and 400 surface points directly from encoder tokens.
- Fast Mesh Fitter: A separate, highly optimized stage fits a MANO or MHR-Hand mesh to the predicted points using a combination of a learned MLP and a single Gauss-Newton optimization step.
- Causal Video Refinement: A 4-layer transformer looks at the last 32 frames to smooth trajectories and reduce "acceleration error" (jitter) in real-time.
- World-Space Tracking: A causal camera model (FAH-camera) combines visual odometry (DPVO) and metric depth estimation (MoGe-2) to place the hands in a global coordinate system.
- Interaction Heads: Optional heads for anatomical part segmentation and hand-object contact probability.
Datasets
The models are trained on a massive "harmonized" mixture of datasets, including HOT3D, COCO-WholeBody, FreiHAND, EgoDex, DexYCB, and InterHand2.6M.
The authors also introduce three new model-assisted stress tests to evaluate performance in difficult conditions:
- Challenging-pose (hard_pose_v1): 175 hands in complex grips and gestures.
- Low-light (dark_v1): Real and synthetic dark photos.
- Shadows (shadow_v1): Images with harsh shadows across or near the hand.
Key Insights and Ideas
- Points > Parameters: The "aha moment" is that directly predicted 3D points agree much better with 2D image evidence than points derived from regressed MANO parameters. Fitting a mesh to these "noisy" but well-aligned points results in lower overall error.
- Harmonization Matters: Different datasets use different joint conventions (e.g., ARKit vs. MANO). Standardizing these before training is crucial for reaching state-of-the-art performance.
- Contact from Geometry: You don't need to know the object's shape to estimate contact. Hand features and predicted geometry alone are surprisingly effective at predicting which parts of the hand are touching an object.
- Causal vs. Offline: While offline models (looking at the whole video) are usually better, FAH shows that a causal transformer (looking only at the past) can drastically reduce jitter while maintaining interactive speeds.
Key Results
FAH sets new benchmarks for both speed and accuracy:
- Speed: The default FAH-L model runs at 65 FPS on an RTX 4070 Ti SUPER. The smallest FAH-S model reaches 125 FPS. Even on an M3 Pro MacBook, the small model runs at over 30 FPS.
- Accuracy: On the HOT3D benchmark, FAH-L achieves a PA-MPJPE (articulation error) of 4.31 mm, significantly outperforming baselines like HaMeR (9.46 mm) and WiLoR (6.72 mm).
- 2D Alignment: In 2D pose estimation, FAH-L reaches 0.83 [email protected], compared to 0.72 for the best previous baseline.
- Contact Estimation: FAH-L outperforms specialized contact models like HACO on both HOT3D and DexYCB datasets without requiring object meshes at inference time.
Limitations and Future Work
- Camera Drift: World-space accuracy is currently limited more by the camera trajectory estimation than by the hand reconstruction itself.
- Occlusion: Contact estimation remains difficult when the contact points are hidden from the camera's view (which happens in 96% of contact cases in some datasets).
- Future Directions: The authors suggest that integrating even stronger foundation models (like SAM 3) for mask supervision or more robust optical flow models could further push accuracy, though at a higher computational cost.
Summary updated