Abstract
Matching near-infrared (NIR) surveillance images against visible light (VIS) face databases is a critical challenge due to the significant spectral gap between modalities. We present a "plug-and-play" framework that adapts pre-trained VIS face recognition models for NIR-VIS matching without retraining the core engine. Our approach utilizes a patch-based hallucination Convolutional Neural Network (CNN) to transform NIR images into the VIS domain and a low-rank linear embedding to align features in a common subspace. Experimental results on the CASIA NIR-VIS v2.0 benchmark demonstrate that our method significantly improves recognition accuracy, achieving state-of-the-art performance by leveraging existing high-performance VIS-trained models.
Introduction
Most high-performance face recognition systems are trained on massive datasets of visible light (VIS) images. However, surveillance often occurs in low-light conditions using near-infrared (NIR) sensors. Matching an NIR "probe" image to a VIS "gallery" image is difficult because of the spectral gap: the same face looks fundamentally different under different light spectrums.
While one could theoretically train a new deep model on NIR data, NIR datasets are significantly smaller than VIS ones. The authors tackle the challenge of leveraging powerful, pre-existing VIS-trained models for NIR-VIS matching without the costly and data-hungry process of retraining them from scratch.
Proposed Framework
The authors propose a two-pronged approach that "wraps" around an existing Deep Neural Network (DNN), as illustrated in the system diagram below.
1. Cross-spectral Hallucination (Input Side)
A Convolutional Neural Network (CNN) is used to transform NIR images into "hallucinated" VIS images. Instead of processing the whole face at once, a patch-based approach (40x40 pixels) is employed to learn local mappings. To ensure the final image retains the sharp details of the original NIR source, a blending post-processing step mixes the CNN's output with the original NIR luminance.
2. Low-rank Embedding (Output Side)
Even with hallucination, features extracted by a VIS-trained DNN from an NIR image may not align perfectly with VIS features in high-dimensional space. The authors append a simple linear transformation (a low-rank embedding) to the end of the DNN. This transform is learned by minimizing the nuclear norm, forcing features from the same person into a low-dimensional subspace while pushing different people's features apart.
Datasets and Evaluation
The framework was evaluated on the CASIA NIR-VIS v2.0 benchmark, which contains 17,580 images of 725 subjects. To facilitate patch-based training, the authors "mined" the dataset to create a secondary dataset of 1.2 million aligned NIR-VIS patch pairs. Additionally, CASIA-Webface was used to train a baseline "VGG-S" model to demonstrate effectiveness on models that have never seen an NIR image.
Key Results and Insights
The researchers tested their approach on three different VIS-trained models (VGG-S, VGG-face, and a commercial off-the-shelf "COTS" model).
- Performance Gains: The VGG-S baseline jumped from 75.04% to 95.72% rank-1 accuracy on the CASIA benchmark when adding both hallucination and embedding.
- State-of-the-Art: Using a COTS model with their framework, they achieved 96.41% accuracy, outperforming previous methods that required complex fine-tuning or specialized architectures.
- Complementary Components: The authors found that hallucinating the input and embedding the output are roughly equally effective when used alone. However, they are complementary—using both together yields the best results.
- Geometric Nature of the Spectral Gap: The success of the low-rank transform suggests that the difference between NIR and VIS features is largely a linear/geometric distortion that can be corrected with a simple matrix multiplication.
Conclusion
This "transfer learning at the input/output" approach demonstrates that massive, complex VIS recognition models do not need to be retrained for cross-modal tasks. By fixing the "entry" and "exit" points, models can be adapted to completely different domains. Future work will address patch-based artifacts, such as distortions around glasses, and investigate automated parameter tuning for the blending process.
Summary updated