A Large-Scale, Time-Synchronized Visible and Thermal Face Dataset

Domenick Poster, Matthew Thielke, Robert Nguyen, Srinivasan Rajaraman, Xing Di, Cedric Nimpa Fondje, Vishal M. Patel, Nathaniel J. Short, Benjamin S. Riggan, Nasser M. Nasrabadi, Shuowen Hu

TL;DR

This paper introduces the ARL-VTF, the largest publicly available dataset of synchronized visible and thermal face images to date, featuring over 500,000 images from 395 subjects. This resource aims to bridge the gap in thermal face data for research and algorithm development, enabling advancements in areas like low-light recognition and cross-modal face verification.

What Problem Does This Paper Address?

The field of face recognition and analysis has largely relied on visible spectrum imagery. However, thermal imaging, which captures heat emissions, offers significant advantages in low-light, nighttime, and even some medical applications. Despite this potential, the availability of large-scale, synchronized visible and thermal face datasets has been a major bottleneck. Existing datasets are often limited in size, resolution, synchronization, or the variety of conditions captured. This scarcity hinders the development and robust evaluation of algorithms that can leverage both modalities, such as those for domain adaptation, cross-modal synthesis, and robust recognition in challenging environments.

Methods

The core contribution of this paper is the creation and release of the ARL-VTF dataset. The data was collected using a modern long-wave infrared (LWIR) camera paired with a stereo setup of three visible spectrum cameras, ensuring time-synchronization. The dataset captures variations in expressions, head pose, and the presence of eyewear. Beyond the dataset itself, the paper also presents benchmark results for two key tasks: thermal face landmark detection and thermal-to-visible face verification. For landmark detection, they evaluated a Deep Alignment Network (DAN), a multi-stage CNN known for its performance in this area. For face verification, they benchmarked a domain-adaptive feature extractor and three Generative Adversarial Network (GAN)-based synthesis approaches, comparing their performance against naive and ground-truth baselines.

Datasets

The primary contribution is the ARL-VTF dataset, which is a newly introduced benchmark.

  • Size: Over 500,000 images from 395 subjects.
  • Modalities: Synchronized visible (stereo RGB and monochrome) and LWIR thermal imagery.
  • Variability: Captures baseline, expression, and pose sequences, with an additional condition for subjects wearing glasses.
  • Annotations: Includes face bounding boxes, 6 facial landmarks, head pose, and eyewear information.
  • Key Differentiator: It is presented as the largest publicly available collection of paired visible and thermal face images with time-synchronization.

The paper also references and compares its dataset against several existing thermal face datasets, including UND [5], IRIS [1], IRIS-M3 [4], UH [3], NVIE [34], KTFE [25], Carl [9], ULFMT [12], ARL-MMFD [15][40], Eurocom [22], and RWTH [20].

Key Insights and Ideas

  • The critical need for synchronized, large-scale thermal data: The paper strongly emphasizes that the scarcity of such datasets is a major impediment to progress in thermal face analysis. The ARL-VTF dataset directly addresses this.
  • Thermal imaging's potential beyond low-light: While often associated with night vision, the paper highlights applications in healthcare, suggesting broader utility for thermal face analysis.
  • Challenges in cross-modal face verification: The benchmark results reveal that while significant progress has been made, pose variations and the presence of glasses (especially in thermal imagery due to heat absorption) remain substantial challenges for current state-of-the-art methods.
  • Glasses as a unique challenge in thermal: The paper notes that while glasses can sometimes improve performance bounds in frontal views (possibly due to distinct visual cues), they also introduce occlusion and heat absorption effects that complicate thermal-to-visible matching.

Key Results

  • Landmark Detection: The Deep Alignment Network (DAN) achieved good performance on frontal images, including those with expressions or glasses. However, performance significantly degraded on the head pose sequence, with a failure rate of 58.68% for yaw angles beyond ±20°.
  • Thermal-to-Visible Face Verification:
    • The "Raw" baseline (direct feature extraction without synthesis) performed poorly, with an AUC of only 61.37%.
  • GAN-based synthesis methods showed significant improvements. The SAGAN method achieved the highest AUC of 99.28% for the baseline sequence.
  • Pose variations had the most detrimental impact on verification performance, with AUC dropping significantly for pose sequences compared to baseline.
  • Expression variations also slightly degraded performance across most methods.
  • The presence of glasses in one image but not the other posed a notable challenge, particularly in the thermal domain.

Limitations and Future Work

  • Pose Sensitivity: The paper explicitly acknowledges that pose is a significant challenge for both landmark detection and thermal-to-visible face verification. The performance degradation on off-pose images is a key limitation.
  • Glasses as a Cross-Modal Challenge: The difficulty in matching images with and without glasses, especially when the thermal image has glasses, is highlighted as an area needing further research.
  • Future Work: The authors suggest that more advanced methods are needed to specifically address pose, expression, and occlusion variations for cross-modal synthesis and verification. The dataset's structure also allows for exploring other cross-modal tasks beyond verification, such as 3D model learning and domain adaptation.

Summary updated

Abstract

Thermal face imagery, which captures the naturally emitted heat from the face, is limited in availability compared to face imagery in the visible spectrum. To help address this scarcity of thermal face imagery for research and algorithm development, we present the DEVCOM Army Research Laboratory Visible-Thermal Face Dataset (ARL-VTF). With over 500,000 images from 395 subjects, the ARL-VTF dataset represents, to the best of our knowledge, the largest collection of paired visible and thermal face images to date. The data was captured using a modern long wave infrared (LWIR) camera mounted alongside a stereo setup of three visible spectrum cameras. Variability in expressions, pose, and eyewear has been systematically recorded. The dataset has been curated with extensive annotations, metadata, and standardized protocols for evaluation. Furthermore, this paper presents extensive benchmark results and analysis on thermal face landmark detection and thermal-to-visible face verification by evaluating state-of-the-art models on the ARL-VTF dataset.
BibTeX
@inproceedings{poster2021large,
  author = {Domenick Poster and Matthew Thielke and Robert Nguyen and Srinivasan Rajaraman and Xing Di and Cedric Nimpa Fondje and Vishal M. Patel and Nathaniel J. Short and Benjamin S. Riggan and Nasser M. Nasrabadi and Shuowen Hu},
  title = {{A Large-Scale, Time-Synchronized Visible and Thermal Face Dataset}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=OiDPY0-yPfZ},
}
Figure