Abstract
Heterogeneous Face Recognition (HFR) involves matching face images across different modalities, such as near-infrared (NIR) or sketches, against a gallery of standard visual (VIS) photographs. This task is complicated by a significant modality gap and the limited availability of paired cross-domain datasets. This paper introduces a robust two-step framework to address these challenges. The first step employs an Image Translation Module that utilizes a CycleGAN integrated with a Siamese network to perform style transfer while preserving identity-specific content. The second step, the Feature Learning Module, fine-tunes a pre-trained ResNet-101 backbone on the translated images using Additive Angular Margin Loss (ArcFace). Our approach achieves state-of-the-art results on the CUFSF and CASIA NIR-VIS 2.0 benchmarks, demonstrating superior performance in bridging the domain gap for face recognition.
Introduction
Face recognition technology has achieved remarkable success in intra-modal scenarios, particularly with high-quality visual images. However, Heterogeneous Face Recognition (HFR) remains a formidable challenge. HFR is critical for applications such as law enforcement, where a forensic sketch must be matched against a database of mugshots, or night-time surveillance using infrared imaging.
The difficulty in HFR stems from two primary factors:
- The Modality Gap: Images from different sensors or artistic renderings (sketches) have vastly different statistical properties, making direct comparison in the pixel or feature space highly inaccurate.
- Data Scarcity: Large-scale visual datasets contain millions of images, whereas HFR datasets are typically small, leading to overfitting when training deep models from scratch.
This paper proposes a "translation-then-recognition" strategy. By converting non-visual images into the visual domain, we can leverage the rich feature representations learned by models trained on massive visual datasets.
Methodology
The proposed framework is divided into two main stages: image-to-image translation and discriminative feature learning.
Step 1: Image Translation Module
The Image Translation Module aims to bridge the modality gap at the image level by generating "pseudo-photos" from non-visual inputs.
- Preprocessing Chain (PC): To handle variations in illumination and noise, input images are processed using gamma correction, Difference of Gaussian (DoG) filtering, and histogram equalization. This step narrows the intensity gap between domains before they enter the neural network.
- Identity-Preserving CycleGAN: We utilize a CycleGAN architecture for unpaired image-to-image translation. To ensure that the translated pseudo-photo retains the identity of the original subject, we incorporate a Siamese network. By applying a contrastive loss between the embeddings of the source and translated images, the model is forced to preserve facial structure and unique identity markers during the style transfer process.
Step 2: Feature Learning Module
After translation, the resulting pseudo-photos are used to train a recognition model that can effectively match probes against a visual gallery.
- Backbone Architecture: We employ a ResNet-101 model pre-trained on the MS-Celeb-1M dataset. This provides a strong starting point for feature extraction.
- Fine-tuning with ArcFace: The backbone is fine-tuned using a combination of original visual images and the generated pseudo-photos. We use the Additive Angular Margin Loss (ArcFace) to enhance the discriminative power of the features. ArcFace effectively increases the geodesic distance between different classes on a hypersphere, making the model more robust to cross-domain variations.
Experimental Results
The framework was evaluated on two major HFR datasets to test its versatility across different non-visual modalities.
Sketch-to-VIS Recognition (CUFSF)
The CUHK Face Sketch FERET (CUFSF) dataset contains 1,194 subjects. Our method achieved a 98.70% Rank-1 accuracy. In comparison, a standard ResNet-101 model without the translation step achieved only 63.20%, highlighting the necessity of the translation module. Our results also surpassed previous state-of-the-art methods like G-HFR (96.04%).
NIR-to-VIS Recognition (CASIA NIR-VIS 2.0)
On the CASIA NIR-VIS 2.0 dataset, which features significant pose and expression variations, our framework achieved a 99.40% Rank-1 accuracy and a verification rate of 98.74% at FAR=0.1%. This performance exceeds the previous state-of-the-art, W-CNN (98.70%).
Ablation Analysis
Ablation studies confirmed the effectiveness of the Preprocessing Chain (PC). Applying PC alone improved accuracy by approximately 2.66% on the sketch dataset, proving that traditional image processing techniques can significantly stabilize and enhance deep learning-based translation.
Conclusion
We have presented a two-step framework for HFR that effectively addresses the modality gap and data scarcity issues. By combining identity-preserving image translation with high-margin feature learning, our method sets new performance benchmarks on major datasets. Future research will focus on developing unified multi-modal translation models that can handle multiple non-visual domains within a single architecture.
Summary updated