End-to-End Detection and Pose Estimation of Two Interacting Hands

Donguk Kim, Kwang In Kim, Seungryul BaekICCV 2021 · OpenReview

Abstract

While 3D hand pose estimation has matured for single-hand scenarios, the accurate estimation of two closely interacting hands remains a significant challenge due to severe self-occlusion and complex inter-dependencies. We present a novel end-to-end framework that jointly detects hands and estimates their 3D poses. Our approach leverages the statistical interdependence between interacting hands and employs a GAN-based discriminator to enforce physical plausibility, preventing impossible configurations like intersecting fingers. Experiments on major benchmarks demonstrate that our unified model significantly outperforms existing state-of-the-art methods in complex interaction scenarios.

The Challenge of Interacting Hands

In real-world applications, hands frequently interact, touch, or overlap. This "interacting hands" scenario introduces two primary hurdles: occlusion and physical impossibility. When hands are intertwined, one often hides critical joints of the other, leading to missing data. Traditional discriminative models often fail here, producing "ghostly" artifacts where fingers from different hands appear to pass through one another.

Most prior research either focuses on isolated single hands or requires pre-cropped images, failing to address the full pipeline from raw image to 3D coordinates. This paper proposes a unified, discriminative CNN-based approach to handle both single and interacting hand scenarios in a single pass.

Methodology

Our architecture is designed to treat interacting hands as a joint entity. The pipeline consists of several specialized modules:

  1. Hand Detection ($f^{Det}$): Utilizing a ResNet-50 backbone with a Faster R-CNN-style proposal network, the system detects hands and classifies them into three distinct categories: 'left', 'right', or 'interacting'. This allows the system to route complex cases to specialized sub-networks.
  2. Joint Pose Estimation ($f^{HPE}$): For interacting hands, the model shares early layers to capture the statistical interdependence between the two hands. We validated this interdependence using the Hilbert-Schmidt Independence Criterion (HSIC), proving that the position of one hand provides significant information about the other.
  3. Visibility-Guided Enhancement: The network explicitly predicts joint visibility. A "Visibility-guided Heatmap Enhancer" uses this to weight 2D heatmaps, forcing the model to infer occluded joint positions from more reliable, visible features.
  4. GAN-type Discriminator: To ensure anatomical realism, we employ a Generative Adversarial Network (GAN). The discriminator learns the distribution of valid hand configurations and penalizes the pose estimator for suggesting physically impossible poses.
Figure 5: The end-to-end pipeline showing detection boxes for left, right, and interacting hands alongside the resulting pose overlays.

Experimental Results

The model was evaluated on three major benchmarks: InterHand2.6M (million-scale real-world images), Ego3D (synthetic egocentric views), and the Tzionas Dataset (real-world interactions).

Our method achieved new state-of-the-art performance across the board:

  • InterHand2.6M: Improved Mean Per Joint Position Error (MPJPE) to 12.08mm, a 4.05% improvement over previous leading systems.
  • Ego3D: Reduced 3D End-Point Error (EPE) from 12.20mm to 11.63mm.
  • Tzionas: Achieved a 2D EPE of 12.42 pixels, demonstrating superior generalization to real-world interactions.
Figure 4: 3D visualizations of the estimated skeletons for both separated and closely interacting hand configurations.

Qualitatively, the model excels at "untangling" tightly intertwined fingers where previous models typically produced distorted or overlapping skeletal structures. The GAN discriminator effectively acts as a learned physics engine, ensuring the output remains anatomically plausible even under heavy occlusion.

Conclusion and Future Work

This work represents the first end-to-end network capable of detecting and estimating the pose of two closely interacting hands. By treating interaction as a joint estimation problem and enforcing physical constraints through adversarial learning, we have significantly pushed the boundaries of hand pose estimation.

Future research will focus on integrating the MANO 3D hand mesh model to estimate full surface geometry and skin deformation, as well as utilizing recurrent neural networks (RNNs) to improve temporal consistency in video sequences.

Summary updated

Abstract

Three dimensional hand pose estimation has reached a level of maturity, enabling real-world applications for single-hand cases. However, accurate estimation of the pose of two closely interacting hands still remains a challenge as in this case, one hand often occludes the other. We present a new algorithm that accurately estimates hand poses in such a challenging scenario. The crux of our algorithm lies in a framework that jointly trains the estimators of interacting hands, leveraging their inter-dependence. Further, we employ a GAN-type discriminator of interacting hand pose that helps avoid physically implausible configurations, e.g. intersecting fingers, and exploit the visibility of joints to improve intermediate 2D pose estimation. We incorporate them into a single model that learns to detect hands and estimate their pose based on a unified criterion of pose estimation accuracy. To our knowledge, this is the first attempt to build an end-to-end network that detects and estimates the pose of two closely interacting hands (as well as single hands). In the experiments with three datasets representing challenging real-world scenarios, our algorithm demonstrated significant and consistent performance improvements over state-of-the-arts.
BibTeX
@inproceedings{kim2021end,
  author = {Donguk Kim and Kwang In Kim and Seungryul Baek},
  title = {{End-to-End Detection and Pose Estimation of Two Interacting Hands}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=5_z5FXHdjb},
}
Figure