WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild

Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, Stefanos Zafeiriou

TL;DR

WiLoR (Wild Localization and Reconstruction) is a real-time, full-stack pipeline for multi-hand localization and 3D hand mesh reconstruction in unconstrained environments. By introducing a massive new 2-million-image dataset (WHIM) and a novel multi-scale vertex refinement module, WiLoR achieves state-of-the-art accuracy and remarkable temporal stability in video tracking—all without relying on any explicit temporal modeling components.

What Problem Does This Paper Address?

While 3D hand pose estimation has advanced significantly, most existing state-of-the-art methods operate under a major assumption: they require a tightly cropped image around a single hand. In real-world, "in-the-wild" scenarios, systems must first localize multiple hands under challenging conditions like motion blur, severe occlusions, and diverse lighting before reconstructing them.

Existing hand detectors are either too slow for real-time applications or fail to generalize to complex multi-hand interactions. This poor localization directly degrades downstream 3D reconstruction, causing severe frame-to-frame jittering artifacts in video sequences. WiLoR addresses this gap by co-designing a highly efficient, robust hand detector with an image-aligned 3D reconstruction network.

Qualitative Evaluation of the proposed hand detection network on in-the-wild images.

Methods

WiLoR splits the problem into a two-stage, coarse-to-fine pipeline:

  1. Hand Detection and Localization: The authors deploy a lightweight, single-stage, anchor-free Fully Convolutional Network (FCN) built on a DarkNet backbone and a Path Aggregation Network (PANet). It simultaneously predicts bounding boxes, handedness (left/right), and joint keypoints. Crucially, adding joint keypoint supervision during training significantly hardens the detector against occlusions and blur.
  2. 3D Hand Reconstruction & Multi-Scale Refinement: The cropped hand image is processed by a Vision Transformer (ViT) encoder to regress initial, rough MANO hand model parameters and camera parameters. To fix the spatial misalignments common in direct parameter regression, the authors introduce a Multi-Scale Pose Refinement Module. The rough 3D mesh vertices are projected back onto multi-resolution feature maps derived from the ViT tokens. Features are sampled per-vertex using bilinear interpolation and aggregated to regress final pose and shape residuals.
Overview of the proposed 3D hand pose estimation method.

Datasets

To train a detector capable of handling real-world complexity, the authors introduced WHIM (Wild Hand Images with Multi-annotations), a massive dataset containing over 2 million in-the-wild hand images automatically curated from more than 1,400 YouTube videos. Annotations were generated via an ensemble of state-of-the-art 2D detectors (ViTPose, OpenPose, MediaPipe) combined with a biomechanically constrained 3D fitting optimization.

For 3D hand pose estimation training and evaluation, the authors leveraged a combination of 14 benchmark datasets, including FreiHAND, HO3D, InterHand2.6M, DEX YCB, ARCTIC, and COCO WholeBody.

Key Insights and Ideas

  • Robust Detection as a Temporal Stabilizer: One of the paper's most striking findings is that explicit temporal architectures (like LSTMs or Transformers over time) are not strictly necessary for smooth video tracking. A sufficiently robust, jitter-free frame-by-frame detector naturally stabilizes downstream 3D mesh generation.
  • Vertex-Level vs. Joint-Level Feature Sampling: Instead of just sampling image features at 2D joint projections, WiLoR projects the entire estimated mesh (all vertices) onto multi-scale feature maps. This dense, per-vertex sampling captures fine-grained structural and shape residuals, leading to superior image alignment.
  • Data Scale Trumps Architecture Complexity: By training on the 2M+ images in WHIM, WiLoR's lightweight detector (25MB) easily outperforms models that are over 50x larger but trained on smaller, less diverse datasets.

Key Results

  • Inference Speed & Efficiency: The medium-sized detector (Proposed-M) runs at 138 FPS on an NVIDIA RTX 4090 with a model size of just 25MB. The smaller variant (Proposed-S) reaches 175 FPS. This is roughly 45× faster and 32× smaller than previous leading frameworks like ContactHands.
  • State-of-the-Art 3D Accuracy: On the FreiHAND benchmark, WiLoR achieves a Procrustes-Aligned Mean per Joint Position Error (PA-MPJPE) of 5.5mm, outperforming top models like HaMeR (6.0mm) and Mesh Graphormer (5.9mm). It achieves similar SOTA results on HO3D (7.5mm PA-MPJPE).
  • Drastic Jitter Reduction: In dynamic 3D sequence evaluations on HO3D, WiLoR cut the acceleration-based jitter metric down to 5.92, compared to 20.43 for HaMeR and 38.53 for SimpleHand.
Qualitative Evaluation of proposed and the baseline methods on FreiHAND dataset.

Limitations and Future Work

Despite its strengths, WiLoR can still fail when encountering extreme, highly uncommon finger articulations or heavily crowded environments where hands blend together. Furthermore, because it processes hands individually, it does not explicitly model physical contact constraints during hand-object or hand-hand interactions, which can occasionally result in spatial clipping or unnatural intersections. Finally, the model estimates poses in camera space; the authors suggest that integrating a 3D metric foundational model could extend its capabilities to accurate world-space reconstruction.

Summary updated

Abstract

In recent years, 3D hand pose estimation methods have garnered significant attention due to their extensive applications in human-computer interaction, virtual reality, and robotics. In contrast, there has been a notable gap in hand detection pipelines, posing significant challenges in constructing effective real-world multi-hand reconstruction systems. In this work, we present a data-driven pipeline for efficient multi-hand reconstruction in the wild. The proposed pipeline is composed of two components: a real-time fully convolutional hand localization and a high-fidelity transformer-based 3D hand reconstruction model. To tackle the limitations of previous methods and build a robust and stable detection network, we introduce a large-scale dataset with over than 2M in-the-wild hand images with diverse lighting, illumination, and occlusion conditions. Our approach outperforms previous methods in both efficiency and accuracy on popular 2D and 3D benchmarks. Finally, we showcase the effectiveness of our pipeline to achieve smooth 3D hand tracking from monocular videos, without utilizing any temporal components. Code, models, and dataset are available https://rolpotamias.github.io/WiLoR.
BibTeX
@inproceedings{potamias2025wilor,
  author = {Rolandos Alexandros Potamias and Jinglei Zhang and Jiankang Deng and Stefanos Zafeiriou},
  title = {{WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01143},
}
Figure