Polarimetric Thermal to Visible Face Verification via Self-Attention Guided Synthesis

Xing Di, Benjamin S. Riggan, Shuowen Hu, Nathaniel J. Short, Vishal M. Patel

TL;DR

This paper tackles the challenge of matching faces captured in visible light with those captured in thermal infrared, a difficult task due to significant domain differences. They propose a novel approach using self-attention guided Generative Adversarial Networks (GANs) to synthesize images across these modalities in both directions (thermal-to-visible and visible-to-thermal), achieving state-of-the-art performance on the ARL polarimetric thermal face dataset.

What Problem Does This Paper Address?

Recognizing individuals in varying lighting conditions, especially transitioning from low-light or night-time (thermal) to normal visible light, is a significant challenge in computer vision. This difficulty arises not only from common variations like expression and pose but also from the inherent differences in how visible and thermal sensors capture information. Thermal imaging, particularly polarimetric thermal imaging, offers advantages by capturing geometric and textural details that are not visible in standard thermal images. However, the large domain gap between visible and thermal images makes direct cross-spectrum matching very difficult. Previous methods have focused on synthesizing visible faces from thermal images, but this paper argues that thermal images also contain valuable discriminative information, leading them to propose a bidirectional synthesis approach.

Methods

The core of the proposed method lies in a novel self-attention guided GAN architecture for cross-modal synthesis. Unlike standard GANs, their approach incorporates a self-attention module within both the generator and discriminator. This module, inspired by SAGAN, allows the network to focus on relevant, long-range dependencies within the image, crucial for capturing subtle facial features and identity information.

The synthesis process is bidirectional:

  1. Thermal-to-Visible Synthesis: A generator synthesizes a visible image from a given polarimetric thermal image.
  2. Visible-to-Thermal Synthesis: Another generator synthesizes a thermal image from a given visible image.

These generators are trained using a combination of loss functions:

  • Adversarial Loss: To ensure synthesized images are realistic.
  • Cycle-Consistency Loss: To ensure that synthesizing an image and then synthesizing it back to the original modality reconstructs the original image.
  • Perceptual Loss: To preserve semantic and identity information.
  • Identity Loss: To further enforce identity preservation.
  • L1 Loss: For pixel-level reconstruction accuracy.

After synthesis, deep features are extracted from both the original and synthesized images using a pre-trained VGG-Face network. These features are then fused (averaged) to create a final template for verification. This fusion strategy leverages the complementary information from both modalities.

Datasets

The ARL Polarimetric Thermal Face Dataset is used for both training and evaluation. This dataset consists of polarimetric (Stokes images: S0, S1, S2) and visible images. The paper mentions using Volume I and Volume II of this dataset, with specific protocols for splitting data into training and testing sets.

Key Insights and Ideas

  • Bidirectional Synthesis is Key: The paper's central hypothesis is that synthesizing both visible from thermal and thermal from visible is more effective than just one direction. This acknowledges that thermal images hold unique discriminative power.
  • Self-Attention for Fine-Grained Details: The integration of self-attention mechanisms is crucial for capturing the subtle, long-range dependencies in facial features that are vital for identity recognition, especially across different modalities.
  • Fusion of Original and Synthesized Features: Instead of relying solely on synthesized images, fusing features from the original and synthesized images provides a more robust representation for verification.
  • Polarimetric Advantage: The results consistently show that using polarimetric thermal images yields better performance than conventional thermal (S0) images, highlighting the value of polarization information.

Key Results

The proposed method achieves state-of-the-art performance on the ARL polarimetric thermal face dataset.

  • On Protocol I (Polar-Visible verification), the method achieves an AUC of 93.68% and an EER of 13.46%.
  • On Protocol II (Polar-Visible verification), the method achieves an AUC of 96.41% and an EER of 10.02%. These results significantly outperform several prior state-of-the-art methods, including GAN-VFS, Riggan et al., Mahendran et al., and AP-GAN. The ablation study also demonstrates that the fusion approach significantly improves performance over unimodal synthesis.

Limitations and Future Work

  • Off-the-Shelf Feature Extractor: While the VGG-Face network is effective, it was pre-trained on visible light data. The paper notes that the visible modality outperforms polar in some unimodal evaluations due to this, suggesting that a modality-specific or jointly trained feature extractor could further improve performance.
  • Generalizability: The current evaluation is limited to the ARL polarimetric thermal dataset. Future work could explore the method's effectiveness on other heterogeneous face recognition tasks, such as sketch-to-face matching.
  • Unsupervised Potential: The paper mentions that the framework can be implemented in a completely unsupervised fashion if pairwise supervised data is not available during training, which is a promising direction for real-world applications where labeled data is scarce.

Summary updated

Abstract

Polarimetric thermal to visible face verification entails matching two images that contain significant domain differences. Several recent approaches have attempted to synthesize visible faces from thermal images for cross-modal matching. In this paper, we take a different approach in which rather than focusing only on synthesizing visible faces from thermal faces, we also propose to synthesize thermal faces from visible faces. Our intuition is based on the fact that thermal images also contain some discriminative information about the person for verification. Deep features from a pre-trained Convolutional Neural Network (CNN) are extracted from the original as well as the synthesized images. These features are then fused to generate a template which is then used for verification. The proposed synthesis network is based on the self-attention generative adversarial network (SAGAN) which essentially allows efficient attention-guided image synthesis. Extensive experiments on the ARL polarimetric thermal face dataset demonstrate that the proposed method achieves state-of-the-art performance.
BibTeX
@inproceedings{di2019polarimetric,
  author = {Xing Di and Benjamin S. Riggan and Shuowen Hu and Nathaniel J. Short and Vishal M. Patel},
  title = {{Polarimetric Thermal to Visible Face Verification via Self-Attention Guided Synthesis}},
  booktitle = {ICB},
  year = {2019},
  url = {https://openreview.net/forum?id=0YQHAWBktWQv},
}
Figure