XPose: Towards Extreme Low Light Hand Pose Estimation

Green Rosh, Meghana Shankar, Prateek Kukreja, Anmol Namdev, Pawan Prasad B HWACV 2025

TL;DR

This paper introduces XPose, the first method for estimating 3D hand poses from extremely low-light images captured by standard digital cameras. They also present LLHands, a novel large-scale dataset for this challenging task, and a training methodology that leverages multi-view, well-lit images to guide the low-light pose estimation network.

What Problem Does This Paper Address?

Estimating the 3D pose of a hand from images is a well-studied problem, with deep learning methods achieving impressive accuracy in well-lit conditions. However, this capability drastically degrades in low-light environments due to significant noise and loss of detail. Existing methods for low-light pose estimation often rely on specialized sensors like infrared or Time-of-Flight cameras. This paper tackles the critical gap of performing accurate hand pose estimation from standard RGB cameras in extremely low-light conditions (down to <1 lux), a scenario where even human vision struggles. This is crucial for applications like augmented reality, robotics, and accessibility features on mobile devices.

Methods

The core of XPose lies in its innovative training strategy and data processing.

  • Linear Domain Processing: Instead of working with the standard non-linear sRGB images, the authors process raw camera images in the linear domain. This is motivated by the observation that image processing tasks like denoising are often more effective in the linear domain, and it helps mitigate artifacts introduced by camera ISPs in low light.
  • Multi-View Well-lit Guidance: To overcome the inherent ill-posedness and noise in single low-light images, XPose leverages paired well-lit and multi-view images from their new dataset.
  • Well-lit View Guidance: A pre-trained well-lit hand pose estimator (Nw) is used to distill its learned rich feature representations into the low-light estimator (Nl). This is achieved using a Spatial Transformer Network (STN) to learn feature-level alignment between low-light and well-lit features, followed by a loss function (Lwvg) to minimize their distance. This is combined with a standard camera-coordinate loss (Lcam) using ground-truth labels.
  • Multi-View Guidance: To ensure consistency across different views of the same hand pose, several loss functions are introduced:
  • World Consistency Loss (Lwgt, Lwwl): Enforces consistency of predicted poses in a collective world coordinate system, both with ground truth and with predictions from well-lit images.
  • Pixel Reprojection Loss (Lpr): Ensures consistency in the 2D pixel space by reprojecting estimated 3D poses and comparing them to ground-truth 2D projections.
  • Network Architecture: The backbone for both the low-light and well-lit estimators is a CPN (Cascaded Pyramid Network) [9], featuring a ResNet50 backbone, GlobalNet, and RefineNet modules for keypoint estimation.

Datasets

  • LLHands Dataset: The authors introduce the first large-scale dataset specifically for extreme low-light hand pose estimation. It comprises approximately 120,000 images captured from 10 subjects. Key features include:
  • Both low-light (<1 lux) and well-lit (1000 lux) images.
  • Images in both linear demosaiced and non-linear sRGB domains.
  • Paired multi-view images for each capture.
  • Accurate 2D and 3D hand pose labels, obtained through a novel automated labeling pipeline using RANSAC and multi-view triangulation from well-lit images.

Key Insights and Ideas

  • Linear Domain is Crucial: Processing images in the linear domain, rather than the standard sRGB, significantly improves performance in extreme low-light conditions by avoiding ISP artifacts and simplifying processing.
  • Guidance is Non-Negotiable: Single low-light images are insufficient for accurate hand pose estimation. Leveraging paired well-lit and multi-view data during training is essential to provide the necessary supervision and regularization.
  • Feature-Level Alignment Matters: Simply distilling features from well-lit to low-light images isn't enough due to minor misalignments. Using an STN to learn feature alignment is a key enabler for effective knowledge transfer.
  • Multi-View Consistency is Powerful: Enforcing consistency across multiple views and lighting conditions in both world and pixel space significantly boosts robustness and accuracy.

Key Results

  • State-of-the-Art Performance: XPose achieves a significant improvement in MPJPE (Mean Per Joint Position Error) on the LLHands dataset. It is approximately 12x better than existing methods without pre-processing and ~9x better than methods using a low-light enhancement pre-processor.
  • Dataset Impact: Training on the proposed linear low-light dataset outperforms training on non-linear low-light images by ~1.5x, highlighting the importance of the linear domain.
  • Ablation Studies: The authors demonstrate that their proposed method, incorporating STN, well-lit view guidance, and multi-view guidance, yields the best results, improving MPJPE by ~0.7 compared to a baseline without these components.
  • Computational Efficiency: XPose is faster than other state-of-the-art methods in terms of MACs/pixel.

Limitations and Future Work

  • Dataset Size: While 120K images is substantial, the authors might explore expanding the dataset further with more subjects and diverse scenarios.
  • Generalization: The paper focuses on extreme low-light conditions. Future work could investigate performance across a wider range of lighting conditions and explore generalization to different camera models.
  • Real-time Performance: While XPose is computationally efficient, achieving real-time performance on resource-constrained devices (like smartphones) might require further optimization.
  • Occlusion and Complex Interactions: While the dataset includes various hand poses, further research could focus on improving robustness to severe occlusions and complex hand-object interactions in low-light scenarios.

Summary updated

BibTeX
@inproceedings{rosh2025xpose,
  author = {Green Rosh and Meghana Shankar and Prateek Kukreja and Anmol Namdev and Pawan Prasad B H},
  title = {{XPose: Towards Extreme Low Light Hand Pose Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure