Updating thermal imaging dataset of hand gestures with unique labels

Sreenivasa Reddy Yeduri, Daniel Skomedal Breland, Om Jee Pandey, Linga Reddy CenkeramaddiData 2022 · DOI

TL;DR

This paper introduces an updated, higher-resolution thermal imaging dataset for hand gesture recognition, featuring 160x120 pixel images captured under various conditions. This dataset aims to improve the robustness and accuracy of machine learning models for gesture recognition, particularly in environments where visible light is unreliable.

What Problem Does This Paper Address?

Accurate hand gesture recognition is crucial for human-computer interaction, sign language interpretation, and assistive technologies. While visible light cameras are common for this task, they are sensitive to lighting conditions and can be occluded. Thermal imaging offers a promising alternative as it's independent of ambient light and can capture subtle temperature variations associated with hand movements. However, existing thermal imaging datasets for hand gestures have been limited by low resolution and a lack of diversity in capture conditions. This paper addresses this gap by providing an updated dataset with higher resolution and more varied capture scenarios, enabling the development of more robust and versatile thermal-based gesture recognition systems.

Methods

The authors utilized a FLIR Lepton 3.5 thermal camera module, paired with a Purethermal 2 breakout board and a Raspberry Pi 4 Model B for data acquisition. This setup allowed for the capture of thermal images at a resolution of 160x120 pixels. A Python script, leveraging the flirpy library for camera control and OpenCV and matplotlib for image processing and saving, managed the data collection process. The script was designed to capture images based on user inputs, allowing for controlled acquisition of multiple frames per gesture. The captured images were stored in both color (plasma) and grayscale formats.

Datasets

The paper presents an updated thermal imaging dataset of hand gestures. This new dataset builds upon a previous low-resolution dataset.

  • New Dataset:
    • Resolution: 160x120 pixels
    • Total Images: 14,400
    • Image Formats: Color (plasma) and Grayscale (PNG)
    • Number of Gestures: 10 (representing digits 0-9)
    • Number of Persons: 30
    • Images per Gesture per Person: 24
    • Capture Conditions: Various orientations of the hand and different lighting conditions (though thermal imaging is light-independent).
  • Dataset Size: 154 MB
  • Related Datasets Mentioned:
    • A previous low-resolution thermal imaging dataset (32x32 pixels) for sign language digits.
  • A dataset for sign language digits.

Key Insights and Ideas

  • Higher Resolution Matters: The move from 32x32 to 160x120 pixel resolution significantly enhances the detail available in thermal hand gesture images, which is crucial for fine-grained gesture recognition.
  • Light Independence is a Key Advantage: Thermal imaging's inherent independence from ambient lighting conditions makes it ideal for applications where visible light cameras struggle, such as in low-light environments or for privacy-preserving systems.
  • Diversity in Capture Conditions is Essential: Including variations in hand orientation and "different lighting conditions" (even though thermal is light-independent, this likely refers to the thermal signature of the environment or the hand itself under varying thermal conditions) helps create a more robust dataset that can generalize better to real-world scenarios.
  • Potential for Improved ML Models: The availability of this higher-quality, more diverse dataset is expected to drive the development of more accurate and efficient machine learning models for thermal-based hand gesture recognition.

Key Results

The paper's primary contribution is the creation and description of the updated dataset. While quantitative performance metrics of machine learning models trained on this dataset are not presented in this specific "Data in Brief" article, the authors highlight the dataset's characteristics:

  • Dataset Size: 154 MB, containing 14,400 images.
  • Resolution: 160x120 pixels, a significant improvement over previous low-resolution datasets.
  • Content: 10 distinct hand gestures, captured from 30 individuals, with 24 images per gesture per person.
  • Formats: Both color (plasma) and grayscale versions of the thermal images are provided.

Limitations and Future Work

  • Limitations:
    • The paper focuses on describing the dataset itself rather than evaluating machine learning models trained on it. Therefore, the direct impact on model performance is not quantified here.
  • While the dataset includes images under "different lighting conditions," the inherent light-independence of thermal cameras means this aspect might be less critical than variations in hand orientation or background thermal properties.
  • Future Work (Implied):
    • The authors suggest that this dataset will be valuable for developing and testing efficient algorithms for thermal imaging-based hand gesture recognition.
  • Further research can explore the use of this dataset to train deep learning models for improved accuracy and robustness in various applications.
  • Investigating the impact of different thermal capture settings and environmental thermal variations on gesture recognition performance would be a logical next step.

Summary updated

Abstract

An update to the previously published low resolution thermal imaging dataset is presented in this paper. The new dataset contains high resolution thermal images corresponding to various hand gestures captured using the FLIR Lepton 3.5 thermal camera and Purethermal 2 breakout board. The resolution of the camera is <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mn>160</mml:mn> <mml:mo>×</mml:mo> <mml:mn>120</mml:mn></mml:mrow> </mml:math> with calibrated array of 19,200 pixels. The images captured by the thermal camera are light-independent. The dataset consists of 14,400 images with equal share from color and gray scale. The dataset consists of 10 different hand gestures. Each gesture has a total of 24 images from a single person with a total of 30 persons for the whole dataset. The dataset also contains the images captured under different orientations of the hand under different lighting conditions.
BibTeX
@inproceedings{yeduri2022updating,
  author = {Sreenivasa Reddy Yeduri and Daniel Skomedal Breland and Om Jee Pandey and Linga Reddy Cenkeramaddi},
  title = {{Updating thermal imaging dataset of hand gestures with unique labels}},
  booktitle = {Data},
  year = {2022},
  doi = {10.1016/j.dib.2022.108037},
}
Figure