Abstract
While 3D hand pose estimation has matured for single-hand scenarios, the accurate estimation of two closely interacting hands remains a significant challenge due to severe self-occlusion and complex inter-dependencies. We present a novel end-to-end framework that jointly detects hands and estimates their 3D poses. Our approach leverages the statistical interdependence between interacting hands and employs a GAN-based discriminator to enforce physical plausibility, preventing impossible configurations like intersecting fingers. Experiments on major benchmarks demonstrate that our unified model significantly outperforms existing state-of-the-art methods in complex interaction scenarios.
The Challenge of Interacting Hands
In real-world applications, hands frequently interact, touch, or overlap. This "interacting hands" scenario introduces two primary hurdles: occlusion and physical impossibility. When hands are intertwined, one often hides critical joints of the other, leading to missing data. Traditional discriminative models often fail here, producing "ghostly" artifacts where fingers from different hands appear to pass through one another.
Most prior research either focuses on isolated single hands or requires pre-cropped images, failing to address the full pipeline from raw image to 3D coordinates. This paper proposes a unified, discriminative CNN-based approach to handle both single and interacting hand scenarios in a single pass.
Methodology
Our architecture is designed to treat interacting hands as a joint entity. The pipeline consists of several specialized modules:
- Hand Detection ($f^{Det}$): Utilizing a ResNet-50 backbone with a Faster R-CNN-style proposal network, the system detects hands and classifies them into three distinct categories: 'left', 'right', or 'interacting'. This allows the system to route complex cases to specialized sub-networks.
- Joint Pose Estimation ($f^{HPE}$): For interacting hands, the model shares early layers to capture the statistical interdependence between the two hands. We validated this interdependence using the Hilbert-Schmidt Independence Criterion (HSIC), proving that the position of one hand provides significant information about the other.
- Visibility-Guided Enhancement: The network explicitly predicts joint visibility. A "Visibility-guided Heatmap Enhancer" uses this to weight 2D heatmaps, forcing the model to infer occluded joint positions from more reliable, visible features.
- GAN-type Discriminator: To ensure anatomical realism, we employ a Generative Adversarial Network (GAN). The discriminator learns the distribution of valid hand configurations and penalizes the pose estimator for suggesting physically impossible poses.
Experimental Results
The model was evaluated on three major benchmarks: InterHand2.6M (million-scale real-world images), Ego3D (synthetic egocentric views), and the Tzionas Dataset (real-world interactions).
Our method achieved new state-of-the-art performance across the board:
- InterHand2.6M: Improved Mean Per Joint Position Error (MPJPE) to 12.08mm, a 4.05% improvement over previous leading systems.
- Ego3D: Reduced 3D End-Point Error (EPE) from 12.20mm to 11.63mm.
- Tzionas: Achieved a 2D EPE of 12.42 pixels, demonstrating superior generalization to real-world interactions.
Qualitatively, the model excels at "untangling" tightly intertwined fingers where previous models typically produced distorted or overlapping skeletal structures. The GAN discriminator effectively acts as a learned physics engine, ensuring the output remains anatomically plausible even under heavy occlusion.
Conclusion and Future Work
This work represents the first end-to-end network capable of detecting and estimating the pose of two closely interacting hands. By treating interaction as a joint estimation problem and enforcing physical constraints through adversarial learning, we have significantly pushed the boundaries of hand pose estimation.
Future research will focus on integrating the MANO 3D hand mesh model to estimate full surface geometry and skin deformation, as well as utilizing recurrent neural networks (RNNs) to improve temporal consistency in video sequences.
Summary updated