Identity-Preserved Human Posture Detection in Infrared Thermal Images: A Benchmark

Yongping Guo, Ying Chen, Jianzhi Deng, Shuiwang Li, Hui ZhouSensors 2023 · OpenReview

TL;DR

This paper introduces a new task called Identity-Preserved Human Posture Detection (IPHPDT) in thermal images, addressing privacy concerns in human pose estimation. They release the IPHPDT dataset, the first benchmark for this task, and establish baseline detectors using state-of-the-art models, achieving promising results.

What Problem Does This Paper Address?

Human pose estimation is valuable for many applications, from action recognition to robotics. However, traditional methods rely on RGB images, which can compromise personal privacy by revealing sensitive environmental details. While thermal imaging offers a privacy-preserving alternative, it presents significant challenges due to its lack of rich texture information, making annotation difficult and existing algorithms less effective. This paper aims to bridge this gap by defining a task that combines the privacy benefits of thermal imaging with the richer information of human posture detection, going beyond simple human detection.

Methods

The core contribution is the formulation of the Identity-Preserved Human Posture Detection (IPHPDT) task. This task sits between standard human detection and full human pose estimation, focusing on identifying and localizing humans while also classifying their posture into one of four categories: standing, sitting, lying, and bending. To tackle this, the authors adapt three state-of-the-art object detection architectures: YOLOF, YOLOX, and TOOD. They modify these models by adding an extra prediction head specifically designed to output the human posture. These adapted models are then trained and evaluated on their newly created dataset.

Datasets

The paper introduces the IPHPDT dataset, the first benchmark specifically for identity-preserved human posture detection in thermal images. This dataset is derived from the existing IPHD dataset, which contains thermal images with bounding box annotations. The authors re-annotated these images to include posture labels for four common human poses. The final IPHPDT dataset comprises 62,010 training images and 13,267 testing images, with a balanced distribution of the four postures.

Key Insights and Ideas

  • Privacy-Preserving Pose Estimation is Crucial: The paper highlights the growing need for computer vision systems that respect privacy, making thermal imaging a compelling modality.
  • Thermal Images are Challenging but Promising: Despite the lack of texture, thermal images offer robustness to lighting and environmental conditions, making them suitable for various applications where privacy is paramount.
  • Posture Detection Adds Significant Value: Moving beyond simple bounding boxes, posture information provides richer semantic understanding for downstream tasks.
  • Posture Detection is Harder than Human Detection: Experimental results indicate that detecting human posture in thermal images is more challenging than simply detecting the human itself, likely due to subtle inter-class differences between poses and potential occlusions.

Key Results

The authors evaluated three baseline detectors (IPH-YOLOF, IPH-YOLOX, and IPH-TOOD) on the IPHPDT dataset.

  • Overall Performance: The IPH-TOOD model generally performed best, achieving a mean average precision (mAP) of 70.4% for the composite task of detecting both human category and posture.
  • Posture-Specific Performance: Standing and lying postures were detected with higher accuracy (mAPcp above 70%) compared to sitting and bending postures (below 70%). This is attributed to the larger amount of training data for standing and fewer intra-class variations for both standing and lying.
  • Qualitative Results: The detectors performed well on clear images with minimal occlusion but struggled with complex scenarios involving occlusion, background clutter, and low contrast, sometimes misclassifying objects or failing to detect postures.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Limited Posture Categories: The current dataset only includes four common postures. Future work could expand to include more posture types for applications requiring finer-grained analysis.
  • Simpler Network Architectures: The proposed baseline detectors use relatively simple network architectures. Exploring more advanced architectures could lead to improved performance.
  • Inter-Task Counteracting: The paper notes a potential counteracting effect between localizing humans and identifying their postures, especially when combined into a single task. Future research should focus on methods to mitigate this.
  • Challenging Scenarios: The models still face difficulties in challenging conditions like occlusion and low contrast. Developing more robust methods for these scenarios is a key area for future work.

Summary updated

Abstract

Human pose estimation has a variety of real-life applications, including human action recognition, AI-powered personal trainers, robotics, motion capture and augmented reality, gaming, and video surveillance. However, most current human pose estimation systems are based on RGB images, which do not seriously take into account personal privacy. Although identity-preserved algorithms are very desirable when human pose estimation is applied to scenarios where personal privacy does matter, developing human pose estimation algorithms based on identity-preserved modalities, such as thermal images concerned here, is very challenging due to the limited amount of training data currently available and the fact that infrared thermal images, unlike RGB images, lack rich texture cues which makes annotating training data itself impractical. In this paper, we formulate a new task with privacy protection that lies between human detection and human pose estimation by introducing a benchmark for IPHPDT (i.e., Identity-Preserved Human Posture Detection in Thermal images). This task has a threefold novel purpose: the first is to establish an identity-preserved task with thermal images; the second is to achieve more information other than the location of persons as provided by human detection for more advanced computer vision applications; the third is to avoid difficulties in collecting well-annotated data for human pose estimation in thermal images. The presented IPHPDT dataset contains four types of human postures, consisting of 75,000 images well-annotated with axis-aligned bounding boxes and postures of the persons. Based on this well-annotated IPHPDT dataset and three state-of-the-art algorithms, i.e., YOLOF (short for You Only Look One-level Feature), YOLOX (short for Exceeding YOLO Series in 2021) and TOOD (short for Task-aligned One-stage Object Detection), we establish three baseline detectors, called IPH-YOLOF, IPH-YOLOX, and IPH-TOOD. In the experiments, three baseline detectors are used to recognize four infrared human postures, and the mean average precision can reach 70.4%. The results show that the three baseline detectors can effectively perform accurate posture detection on the IPHPDT dataset. By releasing IPHPDT, we expect to encourage more future studies into human posture detection in infrared thermal images and draw more attention to this challenging task.
BibTeX
@inproceedings{guo2023identity,
  author = {Yongping Guo and Ying Chen and Jianzhi Deng and Shuiwang Li and Hui Zhou},
  title = {{Identity-Preserved Human Posture Detection in Infrared Thermal Images: A Benchmark}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=lGVJbpTVh6},
}
Figure