MetricHand: Geometry-Aware Flow Matching for 4D Hand Reconstruction in Metric World Space

Yibo Bai, Likui Zhang, Jinglei Zhang, Zhengbin Pang, Xingshen Song, Xiaoqing Yin, Jinsheng DengarXiv 2026 · arXiv

TL;DR

MetricHand is a novel framework for 4D hand reconstruction that accurately recovers hand motion in real-world metric coordinates from egocentric video. By fusing local depth priors with hand topology and using a geometry-aware flow matching process, it significantly reduces localization errors and produces smoother, more physically plausible trajectories compared to current state-of-the-art methods.

What Problem Does This Paper Address?

Reconstructing hand motion from a first-person (egocentric) perspective is critical for training robots and understanding human actions. However, doing this in "metric world space"—meaning actual meters in the real world rather than just relative to a camera—is notoriously difficult.

Most existing methods rely on weak-perspective projection. This assumes the hand is a relatively flat object at a single distance from the camera. In reality, especially at the close ranges typical of egocentric video, different parts of the hand (like fingertips vs. the wrist) are at significantly different depths. Ignoring this "joint-dependent depth variation" leads to biased estimates where errors in translation, orientation, and finger pose become entangled. This problem worsens during hand-object interactions where parts of the hand are often occluded.

Figure 1: MetricHand: geometry-guided 4D hand reconstruction. (a) Weak-perspective projection can bias translation (left), whereas our method uses depth priors to guide metric translation estimation (right). (b) Quantitative comparisons on HOT3D and ARCTIC under a shared evaluation protocol; higher is better. (c) Qualitative world-space comparisons, with GT shown in green, ours in blue, and HaWoR in orange. Closer alignment with the green GT meshes indicates more accurate hand-pose reconstruction in the world coordinate system

Methods

MetricHand introduces a geometry-aware flow matching framework. Instead of a single-shot regression, it iteratively refines the hand state. Key components include:

  • RGB-D Fusion: The model doesn't just look at pixels; it uses a frozen monocular depth estimator (like Metric3D) to provide a "metric prior." These depth features are fused with RGB features to preserve local spatial structure.
  • Metric Head: A specialized module that refines coarse translation estimates specifically in viewing-ray and log-depth coordinates, helping to resolve the depth ambiguity inherent in 2D images.
  • Topology-Aware Sampling: The system samples geometric and visual features around 21 "joint anchors" based on the hand's anatomical structure. This ensures that even if a joint is hidden, the model can infer its position from the surrounding surface geometry of the hand or the object it's touching.
  • Spatiotemporal Memory Queue: A FIFO queue maintains joint-associated features across video frames, allowing the model to use past observations to "fill in the blanks" during moments of heavy occlusion.
  • Geometry-Aware Flow Matching: The core generative engine uses Diffusion Transformer (DiT) blocks. It takes Gaussian noise and, conditioned on the fused geometric/visual features and temporal memory, "flows" it toward the correct joint state distribution (translation, orientation, MANO articulation, and shape).
Figure 2: Overview of the MetricHand architecture, showing the flow from multi-encoders to geometry-aware flow matching.

Datasets

The authors evaluated MetricHand on several challenging benchmarks for egocentric hand-object interaction:

  • HOT3D: A large-scale dataset of hand-object interactions with high-quality ground truth.
  • ARCTIC: Focuses on dexterous bimanual hand-object manipulation.
  • DexYCB & HO3D: Used for further validation and zero-shot evaluation on real-world videos.

Key Insights and Ideas

  • Depth as a Local Feature: Traditionally, depth is used as a global scalar to "lift" a 2D pose. MetricHand treats depth as a local geometric feature. Knowing the local surface structure around a joint provides powerful cues for articulation that visual features alone might miss.
  • Anchoring to Topology: By associating features directly with anatomical joints (topology-aware sampling), the model becomes much more robust to occlusion. It learns how the geometry should look around a specific joint.
  • Flow Matching for 4D Motion: Moving from direct regression to flow matching allows the model to better handle the complex, multi-modal distribution of human hand motion over time, leading to much smoother (lower acceleration error) results.

Key Results

MetricHand sets a new state-of-the-art across multiple metrics:

  • Accuracy: It reduces camera-space Mean Per Joint Position Error (MPJPE) by 49.3% on HOT3D and 52.9% on ARCTIC relative to the previous leader, HaWoR.
  • Localization: Absolute world-space MPJPE improved by 28.7% (HOT3D) and 21.7% (ARCTIC), while wrist localization error dropped by nearly half.
  • Temporal Fidelity: World-space acceleration error—a measure of how "jittery" the reconstruction is—decreased by 44.2% to 64.3%, indicating much more stable and physically realistic motion.
Figure 3: Qualitative comparison showing MetricHand (blue) aligning much more closely with ground truth (green) than previous methods like HaWoR (orange).

Limitations and Future Work

Despite its strengths, MetricHand has a few dependencies that limit its absolute accuracy:

  • SLAM and Depth Priors: The world-space accuracy is still tethered to the quality of the underlying SLAM (for camera trajectory) and monocular depth estimation. Errors in these external modules propagate to the final hand reconstruction.
  • Global Placement: While local reconstruction is excellent, achieving perfect "global" placement in a large scene remains a challenge due to the inherent difficulty of long-range depth estimation from a single moving camera.

Future work may involve tighter integration between the hand reconstruction and the SLAM/depth estimation processes to allow for mutual refinement.

Summary updated

Abstract

Egocentric hand motion provides physical motion supervision for dexterous manipulation and vision-language-action pretraining. Reliable use of this supervision requires accurate and temporally coherent 4D hand reconstruction in metric world space. Existing methods typically lift framewise estimates produced under weak perspective into world coordinates. By ignoring joint-dependent depth variation at close range, this approximation couples errors in translation, orientation, and articulation that persist after world lifting and worsen under occlusion. We present MetricHand, a geometry-aware flow matching framework for 4D hand reconstruction in metric world space. MetricHand encodes depth priors from hand-object interaction regions into joint-associated representations that preserve local spatial structure and metric scale. Fused with visual features, this geometry guides coarse metric estimation and conditions every step of temporal generation. The model jointly recovers world-space translation, global orientation, MANO articulation, and shape, while structural priors and temporal context support estimation under partial occlusion. MetricHand reduces camera-space MPJPE by 49.3% and 52.9%, absolute world-space MPJPE by 28.7% and 21.7%, and wrist localization error by 48.5% and 37.9% relative to HaWoR, respectively. World-space acceleration error decreases by 44.2% and 64.3%, demonstrating improved temporal fidelity alongside more accurate absolute localization.
BibTeX
@misc{bai2026metrichand,
  author = {Yibo Bai and Likui Zhang and Jinglei Zhang and Zhengbin Pang and Xingshen Song and Xiaoqing Yin and Jinsheng Deng},
  title = {{MetricHand: Geometry-Aware Flow Matching for 4D Hand Reconstruction in Metric World Space}},
  year = {2026},
  eprint = {2611.21060},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21060},
}
Figure