Polarimetric Thermal to Visible Face Verification via Attribute Preserved Synthesis

Xing Di, He Zhang, Vishal M. Patel

Abstract

Thermal to visible face verification is a challenging problem due to the large domain discrepancy between the modalities. Existing approaches either attempt to synthesize visible faces from thermal faces or extract robust features from these modalities for cross-modal matching. In this paper, we take a different approach in which we make use of the attributes extracted from the visible image to synthesize the attribute-preserved visible image from the input thermal image for cross-modal matching. A pre-trained VGG-Face network is used to extract the attributes from the visible image. Then, a novel Attribute Preserved Generative Adversarial Network (AP-GAN) is proposed to synthesize the visible image from the thermal image guided by the extracted attributes. Finally, a deep network is used to extract features from the synthesized image and the input visible image for verification. Extensive experiments on the ARL Polarimetric face dataset show that the proposed method achieves significant improvements over the state-of-the-art methods.

Introduction

Face recognition has achieved remarkable success under visible light conditions. However, performance degrades significantly in low-light or nighttime environments where surveillance systems must rely on thermal cameras. Thermal sensors detect heat emissions rather than reflected light, creating a massive "domain gap" between thermal and standard visible-light images. While polarimetric thermal imaging—which captures the polarization state of heat—provides more geometric and textural detail than standard thermal imaging, matching these modalities remains a significant challenge.

Previous attempts to bridge this gap using Generative Adversarial Networks (GANs) often fail to preserve critical semantic details. For instance, a subject with a mustache in a visible image might be synthesized as clean-shaven from a thermal input, leading to false rejections. This paper introduces the Attribute Preserved Generative Adversarial Network (AP-GAN) to address these semantic inconsistencies.

Methodology

The proposed framework utilizes a three-stage pipeline to perform cross-modal verification:

  1. Attribute Predictor: A pre-trained VGG-Face network is fine-tuned to extract 10 specific facial attributes (e.g., Arched Eyebrows, Big Nose, Mustache, Young) from the target visible image.
  2. AP-GAN Generator: This U-Net-based architecture takes the thermal image as input. To integrate the semantic attributes, the model employs Multimodal Compact Bilinear (MCB) pooling. Unlike simple concatenation, MCB pooling allows for complex, multiplicative interactions between the attribute vector and image feature maps in the latent space.
  3. Discriminator and Loss Functions: A "triplet-pair" discriminator ensures the synthesized image matches the specific attributes. The model is trained using a composite loss function including Adversarial, $L_1$, Perceptual, Identity, and a specialized Attribute Preserving loss.

Datasets and Experimental Setup

The study utilizes the ARL Polarimetric Face Dataset, which contains polarimetric thermal images and corresponding visible images for 60 subjects. The polarimetric data includes Stokes images ($S_0, S_1, S_2$) and Degree of Linear Polarization (DoLP) images.

Comparison of visible and various polarimetric thermal modalities (S0, S1, S2, DoLP, Polar) from the ARL dataset.

Experiments were conducted using images from "Range 1," with 30 subjects for training and 30 for testing. The images were manually annotated with 10 facial attributes to facilitate the training of the attribute-preserving mechanism.

Key Results

The AP-GAN framework demonstrates significant improvements over previous state-of-the-art (SOTA) methods:

  • Verification Accuracy: The method achieved an Area Under the Curve (AUC) of 88.93% for polarimetric-to-visible matching, surpassing the previous SOTA of 85.43%.
  • Standard Thermal Performance: Even when limited to standard thermal data ($S_0$), AP-GAN reached 84.16% AUC, a notable jump from the previous 82.49%.
  • Attribute Sensitivity: When using ground-truth attributes instead of predicted ones, the AUC increased to 91.28%, indicating that the framework's ceiling will rise as attribute prediction technology improves.
  • Qualitative Fidelity: Visual analysis confirms that AP-GAN preserves skin textures and facial hair more accurately than previous GAN-based translation approaches.

Conclusion

By using high-level semantic attributes as a bridge between modalities, AP-GAN effectively mitigates the domain gap in thermal-to-visible face verification. The integration of MCB pooling and attribute-preserving losses ensures that synthesized images remain faithful to the subject's identity. Future work will focus on scaling the system to larger datasets and exploring unsupervised attribute discovery to reduce the need for manual annotation.

Summary updated

Abstract

Thermal to visible face verification is a challenging problem due to the large domain discrepancy between the modalities. Existing approaches either attempt to synthesize visible faces from thermal faces or extract robust features from these modalities for cross-modal matching. In this paper, we take a different approach in which we make use of the attributes extracted from the visible image to synthesize the attribute-preserved visible image from the input thermal image for cross-modal matching. A pre-trained VGG-Face network is used to extract the attributes from the visible image. Then, a novel Attribute Preserved Generative Adversarial Network (AP-GAN) is proposed to synthesize the visible image from the thermal image guided by the extracted attributes. Finally, a deep network is used to extract features from the synthesized image and the input visible image for verification. Extensive experiments on the ARL Polarimetric face dataset show that the proposed method achieves significant improvements over the state-of-the-art methods.
BibTeX
@inproceedings{di2018polarimetric,
  author = {Xing Di and He Zhang and Vishal M. Patel},
  title = {{Polarimetric Thermal to Visible Face Verification via Attribute Preserved Synthesis}},
  booktitle = {BTAS},
  year = {2018},
  url = {https://openreview.net/forum?id=SMGpy_feZX},
}
Figure