The Use of AI for Thermal Emotion Recognition: A Review of Problems and Limitations in Standard Design and Data

Catherine Ordun, Edward Raff, Sanjay Purushotham

Abstract

With the increased attention on thermal imagery for Covid-19 screening, the public sector may believe there are new opportunities to exploit thermal as a modality for computer vision and AI. Thermal physiology research has been ongoing since the late nineties. This research lies at the intersections of medicine, psychology, machine learning, optics, and affective computing. We will review the known factors of thermal vs. RGB imaging for facial emotion recognition. But we also propose that thermal imagery may provide a semi-anonymous modality for computer vision, over RGB, which has been plagued by misuse in facial recognition. However, the transition to adopting thermal imagery as a source for any human-centered AI task is not easy and relies on the availability of high fidelity data sources across multiple demographics and thorough validation. This paper takes the reader on a short review of machine learning in thermal FER and the limitations of collecting and developing thermal FER data for AI training. Our motivation is to provide an introductory overview into recent advances for thermal FER and stimulate conversation about the limitations in current datasets.

Introduction: The Limitations of Visible Light FER

While Facial Emotion Recognition (FER) using standard RGB (visible light) cameras is a mature field, it faces significant hurdles. It cannot "see" internal physiological states like stress or fatigue, it is prone to racial and ethnic biases, and it raises major privacy concerns regarding facial recognition misuse. Thermal imaging, specifically Long-Wave Infrared (LWIR), addresses these by capturing heat patterns correlated with the Autonomic Nervous System (ANS). For instance, a cooling nose tip—a signal invisible to RGB cameras—can indicate stress. Despite this potential, the transition to AI-driven thermal FER is stalled by a lack of high-fidelity, standardized data necessary to train robust models.

Methodology and Technical Approaches

Thermal FER bridges medicine, psychology, and machine learning. Key technical approaches discussed in recent literature include:

  • Physiological Mapping: Linking thermal radiation (governed by the Stefan-Boltzmann law) to specific facial Regions of Interest (ROIs) that correlate with emotional arousal.
  • Vascular Network Extraction: Utilizing anisotropic diffusion to identify blood vessel patterns, which can serve as a "soft" biometric for semi-anonymous identification.
  • AI Architectures: Transitioning from manual feature extraction (e.g., PCA/LDA) to deep learning models, including CNNs, LSTMs for temporal data, and GANs for thermal-to-visible image translation.
  • Multimodal Fusion: Combining RGB and thermal data (score-level fusion) to significantly reduce error rates compared to using either modality alone.
Comparison of visible (RGB), thermal, and physiological mapping for facial emotion recognition across different expressions.

The Dataset Landscape

A comprehensive review of 14 key thermal FER datasets reveals a stark contrast with the RGB domain. While visible-light FER benefits from "in-the-wild" datasets containing millions of images, thermal benchmarks like the Univ. of Notre Dame (UND), Equinox, and Iris datasets remain small (often <100 subjects) and lab-controlled. Newer efforts like the Tufts Face Database and ThermalWorld aim for higher resolution, but the field still lacks diversity in age and ethnicity, which is essential for creating equitable AI tools.

Key Insights: Privacy and Remote Physiology

Thermal imagery offers a "semi-anonymous" modality. It provides enough detail to recognize emotion but lacks the superficial texture (skin tone, fine features) that makes RGB-based recognition invasive and biased. Furthermore, thermal FER enables remote physiology; it can detect "deceit" via forehead temperature spikes or "empathy" through synchronized thermal responses between individuals. However, an "in-the-wild" gap persists because thermal sensors are highly sensitive to ambient temperature, humidity, and even the subject's recent consumption of caffeine or alcohol.

Performance and Results

Research indicates that thermal classifiers can outperform RGB ones on "difficult" emotions like disgust and fear, where visible muscle movements are subtle but physiological heat changes are pronounced.

  • Accuracy Gains: Deep learning models (e.g., Deep Boltzmann Machines) have achieved 62.9% accuracy in predicting valence, outperforming traditional statistical methods by over 5%.
  • Fusion Benefits: Combining thermal and visible imagery has been shown to drop the Equal Error Rate (EER) from ~19% (thermal only) to ~14.6%.
  • Complex Affects: While RGB accuracy for disgust and fear often hovers between 40-50%, thermal-based classifiers have reached true positive rates as high as 89.9% for the same emotions.

Limitations and Future Directions

To move beyond the lab, the community must address several critical data challenges:

  1. Temporal Dynamics: Moving from static images to video sequences to capture the onset and duration of physiological heat changes.
  2. Experimental Standardization: Establishing standards for room temperature, subject rest time, and sensor distance to ensure cross-study comparability.
  3. Sensor Heterogeneity: Developing AI models that are invariant to the varying spectral sensitivities and noise profiles of different thermal cameras (e.g., FLIR vs. NEC).
  4. Demographic Diversity: Expanding data collection to include children, the elderly, and diverse ethnic groups to ensure equitable AI in healthcare and telemedicine.

Summary updated

Abstract

With the increased attention on thermal imagery for Covid-19 screening, the public sector may believe there are new opportunities to exploit thermal as a modality for computer vision and AI. Thermal physiology research has been ongoing since the late nineties. This research lies at the intersections of medicine, psychology, machine learning, optics, and affective computing. We will review the known factors of thermal vs. RGB imaging for facial emotion recognition. But we also propose that thermal imagery may provide a semi-anonymous modality for computer vision, over RGB, which has been plagued by misuse in facial recognition. However, the transition to adopting thermal imagery as a source for any human-centered AI task is not easy and relies on the availability of high fidelity data sources across multiple demographics and thorough validation. This paper takes the reader on a short review of machine learning in thermal FER and the limitations of collecting and developing thermal FER data for AI training. Our motivation is to provide an introductory overview into recent advances for thermal FER and stimulate conversation about the limitations in current datasets.
BibTeX
@misc{ordun2020use,
  author = {Catherine Ordun and Edward Raff and Sanjay Purushotham},
  title = {{The Use of AI for Thermal Emotion Recognition: A Review of Problems and Limitations in Standard Design and Data}},
  year = {2020},
  eprint = {2009.10589},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.10589},
}
Figure