Abstract
Face recognition in unconstrained environments faces significant challenges due to variations in pose, expression, and imaging modalities. We present Unified Multi-Domain Face Normalization (UMFN), a framework that simultaneously addresses intra-domain variations and the cross-domain gap between Visible (VIS), Near-Infrared (NIR), and Sketch images. By reconstructing neutral facial prototypes and employing a feature-decoupling strategy, UMFN generates domain-agnostic features that preserve identity consistency. Experimental results on benchmarks like LAMP-HQ and CASIA NIR-VIS 2.0 demonstrate state-of-the-art performance in heterogeneous face recognition tasks.
Introduction
Standard face recognition systems often struggle with the "messiness" of real-world data, where subjects may exhibit non-frontal poses, varied expressions, or be captured in different spectral domains. Specifically, Heterogeneous Face Recognition (HFR) requires matching images across domains—such as comparing a near-infrared (NIR) night-vision frame or a forensic sketch to a high-quality visible-light (VIS) gallery photo. Existing methods typically specialize in a single type of variation (e.g., pose correction) or a specific domain pair, often failing when multiple degradations occur simultaneously or when the domain gap is substantial, leading to "identity inconsistency."
Methodology
The UMFN framework introduces a unified approach to face normalization by combining Autoencoders and Generative Adversarial Networks (GANs) to reconstruct a clean, neutral "facial prototype."
Feature Decoupling Mapping Network
The core of UMFN is a feature-decoupling strategy. Using a domain classifier and adversarial training, the system strips away domain-specific information (e.g., the stylistic markers of a sketch) from identity-relevant features. This process creates "domain-agnostic" representations that characterize the individual regardless of the capture modality.
Control Masks for Multi-Domain Generation
The decoder is guided by a 4-digit "control mask," allowing the network to perform both in-domain cleanup and cross-domain translation. For instance, an NIR input with a specific expression can be transformed into a normalized VIS prototype. This flexibility enables the model to bridge the gap between VIS, NIR, and Sketch domains within a single architecture.
Joint HFR Network with Contrastive Learning
To ensure identity preservation, UMFN employs a contrastive learning mechanism. The model aligns the features of a generated prototype with those of a real target photo in the feature space. By fusing domain-agnostic features from the original input with discriminative features from the generated prototype, the system maintains high recognition accuracy even if visual artifacts are present in the reconstruction.
Experimental Evaluation
The researchers evaluated UMFN on several diverse benchmarks:
- LAMP-HQ: A large-scale NIR-VIS dataset (56,000+ images) featuring diverse poses and occlusions.
- BUAA NIR-VIS & CASIA NIR-VIS 2.0: Standard benchmarks for infrared-to-visible matching.
- CUFSF: A dataset for matching grayscale photos to sketches.
- LFW & IJB-A: Classic "in-the-wild" datasets used to test generalization in unconstrained environments.
Results and Discussion
UMFN achieved state-of-the-art results across multiple HFR tasks. On the challenging LAMP-HQ dataset, the model reached a 99.1% Rank-1 accuracy, surpassing previous methods like NLPF (98.8%). Furthermore, when used as a pre-processing step for a standard ResNet-18 recognizer, it improved the Verification Rate (VR) on the IJB-A dataset by 5.35%.
Qualitative analysis indicates that UMFN produces more symmetrical and "clean" faces compared to baseline methods like DR-GAN or TP-GAN, particularly when handling heavy occlusions such as sunglasses. However, sketches remain a significant challenge due to their low information density, occasionally resulting in slight local distortions in the generated prototypes.
Conclusion
UMFN provides a robust, unified framework for face normalization that bridges the gap between multiple imaging domains while correcting for pose and expression. By focusing on "prototypes" rather than incremental fixes and utilizing feature decoupling, the model significantly enhances the performance of heterogeneous face recognition systems. Future work will focus on refining generation for low-detail inputs like sketches and extending the framework to additional modalities such as thermal imaging.
Summary updated