WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild

Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, Stefanos Zafeiriou

Abstract

WiLoR (Wild Localization and Reconstruction) is a high-performance, end-to-end pipeline designed for 3D hand pose estimation in complex, real-world environments. By integrating a lightning-fast anchor-free detector with a transformer-based reconstruction module and training on a massive new dataset of 2 million images, WiLoR achieves state-of-the-art accuracy and real-time speeds (up to 175 FPS) while maintaining remarkable temporal stability in video.

Addressing the "Localization Gap"

While 3D hand pose estimation has seen significant advancements, most existing methods struggle when deployed "in-the-wild." Real-world scenes are characterized by multiple hands, heavy occlusions, motion blur, and diverse lighting—challenges that are rarely captured in controlled laboratory datasets.

Furthermore, a significant "localization gap" persists. Most 3D reconstruction models assume they are provided with a perfectly cropped image of a single hand. In practice, the hand detector often becomes the bottleneck; if the detector is slow or produces jittery bounding boxes, the subsequent 3D reconstruction will be unstable. Additionally, directly regressing 3D coordinates from an image often leads to poor alignment, where the 3D mesh fails to precisely match the hand's actual position in the 2D image.

Methodology

WiLoR addresses these challenges through a robust two-stage architecture:

  1. Real-time Localization: The pipeline utilizes a fully convolutional, anchor-free hand detector based on a DarkNet backbone and a Path Aggregation Network (PANet). This stage predicts bounding boxes and handedness (left vs. right) across multiple scales with extreme efficiency, enabling real-time multi-hand tracking.
  2. Transformer-based Reconstruction: Cropped hand images are processed by a Vision Transformer (ViT) to predict initial MANO parameters, a standard parametric model for hand shape and pose.
  3. Refinement Module: To resolve misalignments, the model projects the initial 3D mesh back onto the ViT's feature maps. By sampling local features at each vertex location, the model uses "mesh-aligned" features to predict residuals ($\Delta\theta, \Delta\beta$). This coarse-to-fine approach ensures the final 3D mesh is precisely aligned with the image pixels.
WiLoR demonstrates robust multi-hand localization and 3D reconstruction in a cluttered dining scene.

The WHIM Dataset

To bridge the gap between laboratory settings and reality, the authors introduced WHIM (Wild Hand Images with Millions). This dataset contains over 2 million in-the-wild hand images extracted from more than 1,400 YouTube videos. The content spans diverse activities including cooking, sports, sign language, and DIY projects, providing the variety necessary for true generalization.

Key Insights and Results

WiLoR demonstrates that a stable detector is the foundation for high-quality video reconstruction. By reducing bounding box jitter at the detection stage, the 3D reconstruction becomes naturally smoother over time without requiring complex temporal layers like LSTMs.

Performance Metrics

  • Speed: The "Proposed-S" (small) version reaches 175 FPS on an RTX 4090, while the "Proposed-M" (medium) model runs at 138 FPS.
  • Efficiency: WiLoR is 45x faster and has a 32x smaller model size than previous state-of-the-art detectors like ContactHands.
  • Accuracy: On the FreiHAND benchmark, WiLoR achieved a PA-MPJPE of 5.5mm, setting a new state-of-the-art record.
The model maintains high accuracy and stability even in dynamic, high-motion environments like professional sports.

Limitations and Future Work

While WiLoR sets a new standard for in-the-wild hand capture, it currently relies on the MANO parametric model, which may limit the representation of extremely fine details or non-standard deformations. Additionally, the refinement module adds a second processing pass that may impact deployment on ultra-low-power mobile devices. Future research may extend this framework to more complex hand-object interactions or integrate it into full-body capture systems.

Summary updated

Abstract

In recent years, 3D hand pose estimation methods have garnered significant attention due to their extensive applications in human-computer interaction, virtual reality, and robotics. In contrast, there has been a notable gap in hand detection pipelines, posing significant challenges in constructing effective real-world multi-hand reconstruction systems. In this work, we present a data-driven pipeline for efficient multi-hand reconstruction in the wild. The proposed pipeline is composed of two components: a real-time fully convolutional hand localization and a high-fidelity transformer-based 3D hand reconstruction model. To tackle the limitations of previous methods and build a robust and stable detection network, we introduce a large-scale dataset with over than 2M in-the-wild hand images with diverse lighting, illumination, and occlusion conditions. Our approach outperforms previous methods in both efficiency and accuracy on popular 2D and 3D benchmarks. Finally, we showcase the effectiveness of our pipeline to achieve smooth 3D hand tracking from monocular videos, without utilizing any temporal components. Code, models, and dataset are available https://rolpotamias.github.io/WiLoR.
BibTeX
@inproceedings{potamias2025wilor,
  author = {Rolandos Alexandros Potamias and Jinglei Zhang and Jiankang Deng and Stefanos Zafeiriou},
  title = {{WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01143},
}
Figure