Vista-Morph: Unsupervised Image Registration of Visible-Thermal Facial Pairs

Catherine Ordun, Edward Raff, Sanjay PurushothamIJCB 2023 · OpenReview

TL;DR

This paper introduces Vista Morph, the first unsupervised method for aligning visible and thermal facial images. By leveraging a Vision Transformer (ViT) within a Spatial Transformer Network (STN) and Generative Adversarial Networks (GANs), Vista Morph effectively registers misaligned VT pairs without requiring manual feature engineering or a reference thermal image, significantly improving downstream generative tasks.

What Problem Does This Paper Address?

Visible-thermal (VT) facial pairs are crucial for various biometric applications like facial recognition and emotion analysis. However, a significant challenge arises from the inherent misalignment between visible and thermal images due to differences in camera calibration, positioning, and sensor characteristics. This misalignment, especially in real-world scenarios with varying lighting conditions (hard, no, and low-light), can severely degrade the performance of person re-identification and generative AI models, such as those for visible-to-thermal (V2T) image translation. Existing methods often rely on manual feature matching or supervised approaches, which are time-consuming, labor-intensive, and not scalable to large datasets or diverse conditions.

Methods

Vista Morph tackles the VT image registration problem using an unsupervised, end-to-end approach. The core of the method lies in a novel architecture that combines:

  • Vision Transformer (ViT) as a Localization Network: Unlike traditional methods that use Convolutional Neural Networks (CNNs), Vista Morph employs a ViT within a Spatial Transformer Network (STN). The ViT's self-attention mechanism is adept at capturing spatial information across image patches, which is crucial for learning accurate geometric transformations.
  • Spatial Transformer Network (STN): The STN, powered by the ViT, learns an affine transformation matrix that warps the thermal image to align with the visible image.
  • Generative Adversarial Networks (GANs): Two GANs are used to facilitate the learning process. The first generator (G1) translates visible to thermal, and the second (G2) translates thermal to visible. This bidirectional translation helps the STN learn the scale and correspondence between the two modalities.
  • Fourier Loss: To specifically address the challenges posed by low- and no-light visible conditions, a Fourier Loss is incorporated. This loss operates in the frequency domain, allowing for better alignment of thermal images even when the visible counterpart lacks clear features.

The training involves four flows: visible to fake thermal, thermal to fake visible, STN registration, and then using the registered thermal image to generate a fake visible image. This setup encourages the STN to learn the geometric mapping that preserves identity and structure.

Datasets

The authors evaluated Vista Morph on three VT facial datasets:

  • Carl: A dataset known to include low-light conditions.
  • Army Research Lab (ARL) Devcom Dataset: A standard VT facial dataset.
  • Eurecom Dataset: Used for both T~V and V~T registration experiments.

Additionally, they tested the model's generalizability on the FLIR ADAS street scenes dataset, which contains non-facial objects and varying lighting conditions.

Key Insights and Ideas

  • ViT for Spatial Transformer Networks: The paper highlights the effectiveness of using Vision Transformers as the localization network within STNs for image registration. This move from CNNs to ViTs leverages self-attention for better spatial feature extraction.
  • Unsupervised Registration is Achievable: Vista Morph demonstrates that robust VT facial registration can be achieved without manual feature engineering or a supervised thermal reference, significantly reducing the barrier to entry for using VT data.
  • Fourier Loss for Low-Light: The introduction of a Fourier Loss is a key innovation for handling challenging low- and no-light visible conditions, where traditional intensity-based losses might struggle.
  • Generative Quality Improvement: Registering VT pairs before training generative models (like V2T translation GANs) leads to generated thermal images with improved identity preservation, reduced artifacts, and better perceptual clarity.
  • Robustness to Real-World Variations: Vista Morph shows resilience to geometric transformations such as erasure and vertical flips, making it practical for real-world applications.

Key Results

  • Registration Performance: Vista Morph consistently outperforms the state-of-the-art Nemar baseline across multiple metrics (SSIM, NCC, Mutual Information) on the tested datasets, particularly excelling in challenging low-light scenarios. For instance, on the Carl dataset, Vista Morph showed significant improvements of 5.1%, 105.0%, and 26.8% in SSIM, NCC, and MI, respectively.
  • Generative Task Improvement: Training V2T translation GANs with Vista Morph-registered data resulted in substantial improvements in FID and LPIPS scores compared to using unregistered data or Nemar-registered data. For example, Carl and Eurecom V2T translations showed decreases of -13.9% and -12.4% in FID, respectively.
  • Identity Preservation: Analysis using thermal vessel maps (a biometric feature) demonstrated that generated thermal faces from Vista Morph-registered data better preserve the identity of the original subjects, showing higher PSNR values for vessel map similarity.
  • Non-Facial Domain Generalization: Vista Morph successfully registered non-facial objects in the FLIR ADAS dataset, indicating its broader applicability beyond faces.

Limitations and Future Work

  • V~T Registration Underperformance: The authors acknowledge that Vista Morph underperforms for Visible-to-Thermal (V~T) registration compared to Thermal-to-Visible (T~V). This is noted in Table 5, where Nemar shows better results for V~T translation.
  • Diffusion Model Performance: While GAN results improve with registration, the paper notes that registration does not improve the results of a conditional Denoising Diffusion Probabilistic Model (DDPM) for T2V translation.
  • Qualitative Differences in T2V: For T2V translation, generated diffusion results were qualitatively similar but showed differences in skin color, eye detail, and clothing when using registered data.
  • Future Work: The authors suggest further investigation into the consistency of results for diverse demographic samples and assessment of generated thermal faces by thermal specialists. They also plan to explore the impact of registration on other generative models.

Summary updated

Abstract

For a variety of biometric cross-spectral tasks, Visible-Thermal (VT) facial pairs are used. However, due to a lack of calibration in the lab, photographic capture between two different sensors leads to severely misaligned pairs that can lead to poor results for person re-identification and generative AI. To solve this problem, we introduce our approach for VT image registration called Vista Morph. Unlike existing VT facial registration that requires manual, hand-crafted features for pixel matching and/or a supervised thermal reference, Vista Morph is completely unsupervised without the need for a reference. By learning the affine matrix through a Vision Transformer (ViT)-based Spatial Transformer Network (STN) and Generative Adversarial Networks (GAN), Vista Morph successfully aligns facial and non-facial VT images. Our approach learns warps in Hard, No, and Low-light visual settings and is robust to geometric perturbations and erasure at test time. We conduct a downstream generative AI task to show that registering training data with Vista Morph improves subject identity of generated thermal faces when performing V2T image translation.
BibTeX
@inproceedings{ordun2023vista,
  author = {Catherine Ordun and Edward Raff and Sanjay Purushotham},
  title = {{Vista-Morph: Unsupervised Image Registration of Visible-Thermal Facial Pairs}},
  booktitle = {IJCB},
  year = {2023},
  url = {https://openreview.net/forum?id=7bJFc6QlmF},
}
Figure