Non-invasive thermal comfort perception based on subtleness magnification and deep learning for energy efficiency

Xiaogang Cheng, Bin Yang, Anders Hedman, Thomas Olofsson, Haibo Li, Luc Van GoolarXiv 2018 · arXiv

Abstract

Non-invasive thermal comfort detection is critical for optimizing Heating, Ventilation, and Air Conditioning (HVAC) energy efficiency. This paper introduces NIDL, a system combining Eulerian Video Magnification (EVM) and a 315-layer deep neural network to estimate skin temperature from standard smartphone video. By magnifying subtle, invisible physiological changes in skin texture and color, NIDL achieves a mean temperature estimation error of 0.48°C. This approach offers a scalable solution for personalized HVAC control, potentially reducing building energy consumption while significantly improving occupant comfort.

Introduction

HVAC systems are the primary energy consumers in modern buildings, accounting for nearly 50% of total consumption. Traditionally, these systems are regulated based on standardized environmental setpoints (e.g., a constant 22°C), which ignore the subjective nature of thermal comfort. To optimize efficiency, real-time feedback on occupant comfort is necessary. However, traditional methods like questionnaires are intrusive, and wearable sensors are impractical for daily use. NIDL addresses this gap by using computer vision to detect the incredibly subtle skin variations—such as pore shrinkage or minute color shifts—that signal thermal changes.

Methodology

The NIDL framework (Non-invasive subtleness magnification and Deep Learning) utilizes a two-stage pipeline to bridge the gap between raw video data and physiological state estimation:

  1. Subtleness Magnification: The system employs Eulerian Video Magnification (EVM) to amplify temporal variations in skin texture and color that are invisible to the naked eye. This acts as a "microscope for video," revealing the features necessary for the neural network to learn.
  2. Deep Learning Architecture: The magnified video is processed by a modified Inception V3 neural network. The researchers extended the architecture to 315 layers by adding an average pooling layer and three new fully connected layers to handle the complex regression task of temperature estimation.
System architecture of the NIDL framework integrated into an intelligent building HVAC control loop.

Experimental Setup and Dataset

The study introduced a high-resolution dataset comprising 1.44 million frames of skin variation video from 16 subjects. Participants were recorded using a standard smartphone (1280 x 720) while undergoing thermal stimulus. Ground truth skin temperature was recorded simultaneously using medical-grade iButton sensors (accuracy ±0.125 °C) attached to the back of the hand. The model was evaluated using "leave-one-out" cross-validation to ensure it could generalize to individuals not present in the training set.

Results and Discussion

The NIDL system significantly outperformed the previous state-of-the-art baseline (NIPST), achieving a mean error of 0.4834°C and a median error of 0.3464°C. This represents a 16.28% improvement in accuracy. Crucially, the model demonstrated high stability by significantly reducing "large" errors (those over 1.0°C), which is vital for preventing erratic HVAC behavior.

Comparison of skin temperature estimation and absolute error between NIDL, NIPST, and i-Button ground truth across different test scenarios.

As shown in the performance plots, NIDL (black line) closely tracks the ground truth i-Button data (red line) across various cooling and heating phases, maintaining a much lower absolute error than the NIPST baseline (blue line). This success proves that consumer-grade hardware is sufficient for high-accuracy thermal sensing when paired with appropriate magnification and deep learning techniques.

Conclusion and Future Work

NIDL represents a major step toward non-invasive, personalized climate control. While the current results are promising, future research will focus on increasing demographic diversity (beyond the initial cohort of 16 subjects) and testing the system in "wild" office environments with varying lighting conditions. Additionally, the development of an "individual sensitivity index" will further refine how the system translates skin temperature into specific HVAC control actions.

Summary updated

BibTeX
@misc{cheng2018non,
  author = {Xiaogang Cheng and Bin Yang and Anders Hedman and Thomas Olofsson and Haibo Li and Luc Van Gool},
  title = {{Non-invasive thermal comfort perception based on subtleness magnification and deep learning for energy efficiency}},
  year = {2018},
  eprint = {1811.08006},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.08006},
}
Figure