Rethinking the Domain Gap in Near-infrared Face Recognition

Michail Tarasiou, Jiankang Deng, Stefanos Zafeiriou

Introduction

Heterogeneous Face Recognition (HFR) is a critical technology for low-light environments, such as mobile face unlocking and border security. It requires matching probe images captured in the near-infrared (NIR) spectrum against a gallery of standard visible-light (VIS) photographs. Historically, the research community has operated under the assumption that a massive "domain gap" exists between these two modalities, necessitating complex solutions like image-to-image translation or intricate feature-level alignment.

However, this paper challenges that narrative. The authors argue that the perceived domain gap was largely a byproduct of using small-capacity models. By investigating the capabilities of modern, large-scale neural networks, they reveal that the richness of VIS datasets can be directly leveraged to solve the NIR problem with surprisingly simple tools.

The Physics-Inspired Bridge

A core insight of this work is the spectral relationship between visible light and near-infrared radiation. Standard RGB sensors capture light in three bands: Blue, Green, and Red. As shown in the spectral sensitivity curves, the Red channel has the highest spectral overlap with the NIR spectrum.

Spectral sensitivity across different wavelengths and corresponding face image modalities.

By recognizing this "Physics Bridge," the authors propose Red Channel Augmentation. During the pre-training phase on massive VIS datasets (like MS1Mv3), they randomly replace RGB images with versions where all three channels are copies of the Red channel. This simple augmentation forces the model to learn features that are invariant to the color shifts and lack of chromatic information characteristic of NIR sensors, effectively preparing the model for the NIR domain before it ever sees an actual NIR image.

Proposed Framework: Regularized Subspace Fine-tuning

While large models exhibit remarkable "zero-shot" NIR capabilities, fine-tuning on specific NIR datasets is still necessary to reach peak performance. However, standard fine-tuning on small HFR datasets often leads to "catastrophic forgetting," where the model overfits to the small target set and loses the robust facial representations learned during VIS pre-training.

To address this, the authors introduce a two-stage framework:

  1. Enhanced VIS Pre-training: Utilizing the Red Channel Augmentation described above.
  2. Regularized Subspace Fine-tuning: This stage employs two key strategies to preserve the pre-trained embedding space. First, they use a Subspace Classifier, initializing weights with mean identity embeddings rather than random values. Second, they apply a Regularized Loss by continuing to feed VIS data into the model during NIR fine-tuning. This ensures the model maintains its general facial recognition intelligence while adapting to the specific sensor noise of the NIR data.
The proposed two-stage framework consisting of VIS pre-training and regularized fine-tuning.

Key Results and Insights

The experimental results validate the authors' hypothesis that model scale significantly impacts the domain gap. While smaller models (e.g., MFN) show a drastic performance drop when moving from VIS to NIR, larger models like ResNet-100 maintain over 99% accuracy (at FAR=$10^{-4}$) on benchmarks like CASIA 2.0 and BUAA-VisNir, even in a zero-shot setting.

Furthermore, the proposed regularized fine-tuning approach set new state-of-the-art records across all major HFR benchmarks. On the high-quality LAMP-HQ dataset, the method achieved 99.35% accuracy, outperforming far more complex generative and domain-invariant models. Crucially, these models demonstrated superior cross-dataset robustness, proving they learned generalized facial features rather than just overfitting to specific sensor characteristics.

Conclusion

This research shifts the focus of HFR from complex cross-modal bridging to leveraging the power of scale and simple physical priors. By rethinking the domain gap, the authors demonstrate that the "NIR problem" is largely solvable through high-capacity models and intelligent regularization. Future work will explore how to distill this high-level performance into the smaller, more efficient models required for real-time mobile applications.

Summary updated

Abstract

Heterogeneous face recognition (HFR) involves the intricate task of matching face images across the visual domains of visible (VIS) and near-infrared (NIR). While much of the existing literature on HFR identifies the domain gap as a primary challenge and directs efforts towards bridging it at either the input or feature level, our work deviates from this trend. We observe that large neural networks, unlike their smaller counterparts, when pretrained on large scale homogeneous VIS data, demonstrate exceptional zero-shot performance in HFR, suggesting that the domain gap might be less pronounced than previously believed. By approaching the HFR problem as one of low-data fine-tuning, we introduce a straightforward framework: comprehensive pre-training, succeeded by a regularized fine-tuning strategy, that matches or surpasses the current state-of-the-art on four publicly available benchmarks. Given its simplicity and demonstrably strong performance, our method could be used as a practical solution for adjusting face recognition models to HFR as well as a new baseline for future HFR research. Corresponding training and evaluation codes can be found at https://github.com/michaeltrs/RethinkNIRVIS.
BibTeX
@inproceedings{tarasiou2023rethinking,
  author = {Michail Tarasiou and Jiankang Deng and Stefanos Zafeiriou},
  title = {{Rethinking the Domain Gap in Near-infrared Face Recognition}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
}
Figure