ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Image

Fangqiang Ding, Yunzhou Zhu, Xiangyu Wen, Chris Xiaoxuan Lu

Abstract

In this work, we present ThermoHands, a new benchmark for thermal image-based egocentric 3D hand pose estimation, aimed at overcoming challenges like varying lighting conditions and obstructions (e.g., handwear). The benchmark includes a multi-view and multi-spectral dataset collected from 28 subjects performing hand-object and hand-virtual interactions under diverse scenarios, accurately annotated with 3D hand poses through an automated process. We introduce a new baseline method, TherFormer, utilizing dual transformer modules for effective egocentric 3D hand pose estimation in thermal imagery. Our experimental results highlight TherFormer's leading performance and affirm thermal imaging's effectiveness in enabling robust 3D hand pose estimation in adverse conditions.

Introduction

Hand pose estimation serves as the primary "spatial interaction" engine for modern extended reality (XR) headsets. However, current industry standards rely heavily on RGB or Near-Infrared (NIR) cameras, both of which face significant limitations. RGB systems fail in low-light conditions or when hands are obscured by gloves. While NIR performs better in the dark, it is power-intensive and susceptible to interference from sunlight or other active sensors.

ThermoHands introduces a robust alternative: thermal imaging (Long-Wave Infrared). As a passive sensor, thermal cameras capture heat radiation rather than reflected light, allowing them to function in total darkness and resist sunlight glare. Furthermore, thermal imaging can detect heat transmission patterns through handwear, potentially "seeing" hand structures through gloves where visual light cameras cannot.

The ThermoHands Benchmark

To facilitate research in this area, we present the ThermoHands dataset, a large-scale multi-spectral collection designed to emulate real-world complexity. The dataset comprises approximately 96,000 synchronized frames across 352 sequences, involving 28 subjects.

Overview of the ThermoHands multi-spectral data collection setup and sensor views.

Participants performed 12 distinct hand-object and hand-virtual interactions, such as writing, pouring water, and swiping. Data was captured across five challenging scenarios: standard office environments, total darkness, intense sun glare, kitchen settings, and while wearing gloves. Each frame includes synchronized Thermal, NIR, Depth, and RGB imagery, providing a comprehensive foundation for multi-modal hand pose research.

Methodology and Sensor Platform

The core of our data collection is a custom Head-Mounted Sensor Platform (HMSP). This platform integrates thermal, RGB, NIR, and LiDAR cameras to capture synchronized egocentric data.

The custom Head-Mounted Sensor Platform (HMSP) featuring the thermal camera and LiDAR sensor.

To calibrate these diverse sensors, we developed a specialized "cooled" chessboard. By creating a sharp temperature differential, the board appears clearly across all spectral ranges, ensuring precise spatial alignment.

For the baseline model, we introduced TherFormer, a dual-transformer architecture designed for thermal imagery. It utilizes a Mask-guided Spatial Transformer to focus on hand regions while filtering out background thermal noise (e.g., from electronic devices). A Temporal Transformer then links information across consecutive frames to maintain tracking accuracy during rapid movements or self-occlusions. Ground truth labels were generated via an automated pipeline that fits a MANO statistical hand model to the thermal data using multi-view RGB and depth information.

Key Results and Insights

Experimental evaluations demonstrate that thermal imaging is a superior "all-weather" sensor for hand tracking.

  • Robustness in Adverse Conditions: Under intense sun glare or in total darkness, thermal-based models significantly outperform RGB and NIR counterparts. While NIR error rates spike under direct sunlight, thermal performance remains stable.
  • Seeing Through Gloves: Thermal cameras can detect the structure of a hand through gloves by identifying heat transmission patterns, a feat impossible for standard visual light cameras.
  • Real-Time Efficiency: TherFormer is optimized for mobile XR applications. The single-image version (TherFormer-S) operates at 136 FPS, while the video-based version (TherFormer-V) maintains 52 FPS on modern hardware.

Conclusion

ThermoHands demonstrates that thermal imaging is a viable, power-efficient, and robust solution for egocentric 3D hand pose estimation. While current work focuses on indoor environments, future expansions will explore outdoor settings and more complex thermal interactions, such as contact with hot objects. This benchmark provides the necessary data and baselines to drive the next generation of robust XR interaction systems.

Summary updated

Abstract

In this work, we present ThermoHands, a new benchmark for thermal image-based egocentric 3D hand pose estimation, aimed at overcoming challenges like varying lighting conditions and obstructions (e.g., handwear). The benchmark includes a multi-view and multi-spectral dataset collected from 28 subjects performing hand-object and hand-virtual interactions under diverse scenarios, accurately annotated with 3D hand poses through an automated process. We introduce a new baseline method, TherFormer, utilizing dual transformer modules for effective egocentric 3D hand pose estimation in thermal imagery. Our experimental results highlight TherFormer's leading performance and affirm thermal imaging's effectiveness in enabling robust 3D hand pose estimation in adverse conditions.
BibTeX
@misc{ding2024thermohands,
  author = {Fangqiang Ding and Yunzhou Zhu and Xiangyu Wen and Chris Xiaoxuan Lu},
  title = {{ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Image}},
  year = {2024},
  eprint = {2403.09871},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.09871},
}
Figure