HandTailor: Towards High-Precision Monocular 3D Hand Recovery

Jun Lv, Wenqiang Xu, Lixin Yang, Sucheng Qian, Chongzhao Mao, Cewu Lu

TL;DR

HandTailor is a hybrid 3D hand recovery framework that combines the speed of deep learning with the precision of classical optimization. By using a fast, differentiable "tailor module" to refine neural network predictions, it achieves state-of-the-art accuracy on monocular RGB images while maintaining a negligible computational overhead of just 8ms.

What Problem Does This Paper Address?

Recovering a 3D hand mesh from a single 2D image is a classic "ill-posed" problem in computer vision—there are many possible 3D hand poses that could result in the same 2D projection. Current methods generally fall into two camps:

  1. Learning-based: Fast and robust, but they often suffer from "back-projection inconsistency," where the predicted 3D hand doesn't perfectly align with the pixels in the original image.
  2. Optimization-based: Highly accurate at aligning models to images, but typically too slow for real-time applications.

Furthermore, most models are locked into a specific camera projection model (either perspective or weak perspective), making them difficult to deploy across both controlled laboratory settings and "in-the-wild" internet images where camera parameters are unknown.

Methods

The authors propose a two-part system called HandTailor:

  • The Hand Module: A multi-stage CNN (based on a stacked hourglass architecture) that predicts intermediate 2D keypoints and 3D joint distances. These are fed into an "IKNet" to estimate MANO hand model parameters (pose $\theta$ and shape $\beta$). Crucially, this module is designed to be projection-agnostic; it can switch between perspective and weak perspective projection analytically without needing to retrain the weights.
  • The Tailor Module: This is the core innovation. It takes the "coarse" mesh from the Hand Module and treats it as an initialization for a fast optimization loop. It minimizes an energy function based on four constraints:
  • Joint Location: Ensuring the 3D joints, when projected back to 2D, match the high-precision 2D keypoints predicted earlier.
  • Scale: Correcting for depth/size noise.
  • Unnatural Twist: A geometric constraint to prevent "monster hands" (anatomically impossible poses).
  • Identity: A regularization term to ensure the optimization doesn't drift too far from the initial neural network guess.
Figure 1: HandTailor consists of two components, hand module and tailor module. Hand module predicts the parameters of MANO through the pose and shape branches to generate hand mesh. It takes a multi-stage design and produces 2D keypoints and 3D joints as inter- mediate results. The tailor module optimizes the predicted hand mesh according to several constraints. HandTailor adapts both perspective projection and weak perspective projection.

To keep this fast, the authors implemented the tailor module using JAX, allowing for Just-In-Time (JIT) compilation and automatic differentiation on the CPU.

Datasets

The model was trained and evaluated on three major benchmarks:

  • Rendered Hand Dataset (RHD): A synthetic dataset used for high-precision evaluation.
  • Stereo Hand Pose Tracking Benchmark (STB): A standard real-world dataset.
  • FreiHAND: A large-scale dataset containing diverse hand poses and shapes.

Key Insights and Ideas

  • 2D is More Reliable than 3D: The authors realized that neural networks are much better at predicting 2D keypoints than 3D coordinates. By treating the 2D predictions as a "reliable anchor," they can use optimization to "pull" the 3D mesh into the correct alignment.
  • Analytical Projection Switching: Instead of training two separate models for different camera types, they derived analytical solutions to calculate scale factors ($s$) for weak perspective projection. This allows the same network to work on both calibrated laboratory data and random internet photos.
  • Optimization as a Plugin: The tailor module is "plug-and-play." The authors demonstrated that it could be added to other researchers' existing models (like IKNet or BiHand) to immediately boost their accuracy.

Key Results

  • State-of-the-Art Performance: HandTailor achieved top-tier results on RHD and STB benchmarks. On the RHD dataset, it reached a PCK (Percentage of Correct Keypoints) of 0.874.
  • Precision Boost: The tailor module significantly improved results on strict metrics. For the $AUC_{5-20}$ metric (which measures errors in the tiny 5mm to 20mm range), HandTailor improved the baseline performance from 0.561 to 0.658.
  • Efficiency: Despite adding an optimization step, the tailor module only adds ~8ms of latency on a modern CPU (Intel i7-8700), making it viable for real-time systems.
  • Qualitative Improvement: Visual results show that the tailor module effectively fixes "finger deviation" errors where the 3D mesh fingers previously didn't line up with the fingers in the photo.

Limitations and Future Work

While HandTailor excels at single-hand recovery, the authors note two primary areas for future research:

  • Interacting Hands: The current framework is not yet optimized for scenarios where two hands are touching or overlapping.
  • Hand-Object Interaction: The model does not currently account for the geometry of objects being held, which can cause the hand mesh to "clip" through or fail to grasp objects realistically.

The team plans to address these complex interaction scenarios in future iterations of the framework.

Summary updated

Abstract

3D hand pose estimation and shape recovery are challenging tasks in computer vision. We introduce a novel framework HandTailor, which combines a learning-based hand module and an optimization-based tailor module to achieve high-precision hand mesh recovery from a monocular RGB image. The proposed hand module unifies perspective projection and weak perspective projection in a single network towards accuracy-oriented and in-the-wild scenarios. The proposed tailor module then utilizes the coarsely reconstructed mesh model provided by the hand module as initialization, and iteratively optimizes an energy function to obtain better results. The tailor module is time-efficient, costs only 8ms per frame on a modern CPU. We demonstrate that HandTailor can get state-of-the-art performance on several public benchmarks, with impressive qualitative results on in-the-wild experiments. Code and video are available on our project webpage https://sites.google.com/view/handtailor.
BibTeX
@misc{lv2021handtailor,
  author = {Jun Lv and Wenqiang Xu and Lixin Yang and Sucheng Qian and Chongzhao Mao and Cewu Lu},
  title = {{HandTailor: Towards High-Precision Monocular 3D Hand Recovery}},
  year = {2021},
  eprint = {2102.09244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.09244},
}
Figure