TL;DR
The Hand Visibility Detector treats per-joint hand visibility estimation as a standalone task for the first time, rather than just an auxiliary signal for pose estimation. By attaching a lightweight attention-based head to a frozen, large-scale pretrained hand pose backbone, the model achieves state-of-the-art visibility detection and significantly improves downstream 3D hand pose annotation via multi-view triangulation.
What Problem Does This Paper Address?
In applications like AR/VR and robotics, knowing exactly which parts of a hand are visible versus occluded is vital for assessing the reliability of hand pose estimation (HPE). However, most existing HPE methods blindly output 21-joint positions without explicitly indicating whether a joint is actually visible or hidden behind an object or the hand itself.
While some prior works have modeled occlusion, they typically treat visibility as a secondary, auxiliary signal to boost coordinate accuracy, leaving the quality of the visibility predictions unexamined. This paper introduces per-joint hand visibility estimation as an independent, standalone task, providing a systematic evaluation of how well models can judge what they can and cannot see.
Methods
The authors propose a simple yet highly effective architecture consisting of two main components:
- A Frozen Hand Encoder: Instead of training a feature extractor from scratch, the method leverages the rich structural prior knowledge of large-scale pretrained HPE models (specifically, the Vision Transformer backbones of HaMeR or WiLoR). Crucially, these weights are kept frozen to preserve the model's understanding of hand anatomy and prevent overfitting on smaller visibility datasets.
- A Lightweight Visibility Head: A small network is attached to the backbone to output a visibility confidence score between 0 and 1 for each of the 21 MANO hand joints. It utilizes a Gated Attention Unit (GAU) to model global spatial dependencies across the hand, allowing the model to reason about hidden joints based on the positions of visible ones.
The entire pipeline is trained using a standard binary cross-entropy loss.
Datasets
- HInt Dataset: Used for training and evaluating the visibility detector. It contains 25,273 training frames and 5,374 evaluation frames, featuring diverse, in-the-wild scenes (web images, egocentric videos) with manual per-joint visibility annotations.
- DexYCB, HO3D, and H2O: Three multi-view datasets used to evaluate the model's utility in a downstream 3D hand pose triangulation task.
Key Insights and Ideas
- Anatomy Priors Trump General Vision Priors: Hand-specific pretrained models (like WiLoR and HaMeR) significantly outperform powerful general-purpose foundation models (like DINOv3) at visibility detection. Understanding the underlying geometry of a hand is essential for inferring what is occluded.
- Do Not Fine-Tune the Backbone: Trying to fine-tune the WiLoR backbone specifically for the visibility task caused performance to plummet (mAP dropped from 0.931 to 0.622). Freezing the backbone is critical to maintaining the robust representations learned during large-scale pretraining.
- Global Context is Essential: Removing the Gated Attention Unit (GAU) from the visibility head degraded performance significantly. To know if a finger joint is occluded, the model needs to look at the global context of the hand—such as whether the palm or an object is blocking the view.
Key Results
- State-of-the-Art Visibility Detection: The Hand Visibility Detector achieved an mAP of 0.931 and an F1 score of 0.896 on the HInt dataset, outperforming existing baselines (like Kim et al. and Contact4D) by 3.4 points in mAP.
- Robust to Thresholding: The model's F1 score remains stable (>0.88) across a wide range of binarization thresholds (0.3 to 0.7), showing it is not overly sensitive to hyperparameter tuning.
- Cleaner 3D Annotations: When used to weight multi-view 2D keypoints during 3D triangulation, the visibility scores effectively suppressed low-quality, occluded joint estimates. This reduced the mean 3D reprojection error by up to 10.1%, with the most substantial gains seen on the highly occluded HO3D dataset.
Limitations and Future Work
The primary limitation of the current Hand Visibility Detector is that it operates strictly on single, static frames (monocular images). It does not exploit temporal context, which could lead to flickering or inconsistent visibility predictions across consecutive video frames.
As a next step, the authors plan to extend the framework to video inputs to achieve temporally consistent visibility estimation.
Summary updated