Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands

Ryosei Hara, Masashi Hatano, Rintaro Yanagi, Atsushi Hashimoto, Takuma Yagi, Mariko IsogawaarXiv 2026 · arXiv

TL;DR

The Hand Visibility Detector treats per-joint hand visibility estimation as a standalone task for the first time, rather than just an auxiliary signal for pose estimation. By attaching a lightweight attention-based head to a frozen, large-scale pretrained hand pose backbone, the model achieves state-of-the-art visibility detection and significantly improves downstream 3D hand pose annotation via multi-view triangulation.

What Problem Does This Paper Address?

In applications like AR/VR and robotics, knowing exactly which parts of a hand are visible versus occluded is vital for assessing the reliability of hand pose estimation (HPE). However, most existing HPE methods blindly output 21-joint positions without explicitly indicating whether a joint is actually visible or hidden behind an object or the hand itself.

While some prior works have modeled occlusion, they typically treat visibility as a secondary, auxiliary signal to boost coordinate accuracy, leaving the quality of the visibility predictions unexamined. This paper introduces per-joint hand visibility estimation as an independent, standalone task, providing a systematic evaluation of how well models can judge what they can and cannot see.

Methods

The authors propose a simple yet highly effective architecture consisting of two main components:

  1. A Frozen Hand Encoder: Instead of training a feature extractor from scratch, the method leverages the rich structural prior knowledge of large-scale pretrained HPE models (specifically, the Vision Transformer backbones of HaMeR or WiLoR). Crucially, these weights are kept frozen to preserve the model's understanding of hand anatomy and prevent overfitting on smaller visibility datasets.
  2. A Lightweight Visibility Head: A small network is attached to the backbone to output a visibility confidence score between 0 and 1 for each of the 21 MANO hand joints. It utilizes a Gated Attention Unit (GAU) to model global spatial dependencies across the hand, allowing the model to reason about hidden joints based on the positions of visible ones.

The entire pipeline is trained using a standard binary cross-entropy loss.

Figure 2. Overview of the proposed method. (a) Overall pipeline. Our model outputs only per-joint visibility, shown as green (visible) and red (invisible). Joint positions are provided by ground truth or an off-the-shelf HPE model for visualization only. (b) Architecture of the Visibility Head. A frozen backbone of a pretrained HPE model extracts features, and only the lightweight head is trained.

Datasets

  • HInt Dataset: Used for training and evaluating the visibility detector. It contains 25,273 training frames and 5,374 evaluation frames, featuring diverse, in-the-wild scenes (web images, egocentric videos) with manual per-joint visibility annotations.
  • DexYCB, HO3D, and H2O: Three multi-view datasets used to evaluate the model's utility in a downstream 3D hand pose triangulation task.

Key Insights and Ideas

  • Anatomy Priors Trump General Vision Priors: Hand-specific pretrained models (like WiLoR and HaMeR) significantly outperform powerful general-purpose foundation models (like DINOv3) at visibility detection. Understanding the underlying geometry of a hand is essential for inferring what is occluded.
  • Do Not Fine-Tune the Backbone: Trying to fine-tune the WiLoR backbone specifically for the visibility task caused performance to plummet (mAP dropped from 0.931 to 0.622). Freezing the backbone is critical to maintaining the robust representations learned during large-scale pretraining.
  • Global Context is Essential: Removing the Gated Attention Unit (GAU) from the visibility head degraded performance significantly. To know if a finger joint is occluded, the model needs to look at the global context of the hand—such as whether the palm or an object is blocking the view.

Key Results

  • State-of-the-Art Visibility Detection: The Hand Visibility Detector achieved an mAP of 0.931 and an F1 score of 0.896 on the HInt dataset, outperforming existing baselines (like Kim et al. and Contact4D) by 3.4 points in mAP.
  • Robust to Thresholding: The model's F1 score remains stable (>0.88) across a wide range of binarization thresholds (0.3 to 0.7), showing it is not overly sensitive to hyperparameter tuning.
  • Cleaner 3D Annotations: When used to weight multi-view 2D keypoints during 3D triangulation, the visibility scores effectively suppressed low-quality, occluded joint estimates. This reduced the mean 3D reprojection error by up to 10.1%, with the most substantial gains seen on the highly occluded HO3D dataset.
Figure 4. Qualitative results of visibility estimation. Joint positions are ground truth. Green indicates joints predicted as visible, and red indicates joints predicted as invisible. From top to bottom: self-occlusion, image truncation, and occlusion by objects.

Limitations and Future Work

The primary limitation of the current Hand Visibility Detector is that it operates strictly on single, static frames (monocular images). It does not exploit temporal context, which could lead to flickering or inconsistent visibility predictions across consecutive video frames.

As a next step, the authors plan to extend the framework to video inputs to achieve temporally consistent visibility estimation.

Summary updated

Abstract

Hand Pose Estimation (HPE) is a fundamental technology for various applications such as AR/VR and robotics. In these applications, the visibility of each hand joint in the image is crucial for assessing the reliability of estimation results under occlusion. However, most existing HPE methods output joint positions without explicitly indicating their visibility. Although some methods account for occlusion or visibility, visibility estimation has mainly been used as an auxiliary signal for improving pose estimation. To our knowledge, per-joint hand visibility estimation has not been systematically studied as a standalone task. In this work, we propose Hand Visibility Detector, a model for estimating the visibility of individual hand joints, and present the first systematic investigation of visibility estimation as an independent task. We show that leveraging the prior knowledge of HPE models pretrained on large-scale data as a backbone yields high performance in this task. We further demonstrate the utility of Hand Visibility Detector on a downstream task of 3D hand pose annotation via multi-view triangulation of 2D keypoints, showing that visibility-weighted triangulation reduces reprojection error. Our method is released as a ready-to-use package, and the code and demo are available at https://github.com/ryhara/hand_visibility_detector .
BibTeX
@misc{hara2026hand,
  author = {Ryosei Hara and Masashi Hatano and Rintaro Yanagi and Atsushi Hashimoto and Takuma Yagi and Mariko Isogawa},
  title = {{Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands}},
  year = {2026},
  eprint = {2608.11574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.11574},
}
Figure