UMFN: Unified Multi-Domain Face Normalization for Joint Cross-domain Prototype Learning and Heterogeneous Face Recognition

Meng Pang, Wenjun Zhang, Nanrun Zhou, Shengbo Chen, Hong RaoCVPR 2025 · DOI

Abstract

Face recognition in unconstrained environments faces significant challenges due to variations in pose, expression, and imaging modalities. We present Unified Multi-Domain Face Normalization (UMFN), a framework that simultaneously addresses intra-domain variations and the cross-domain gap between Visible (VIS), Near-Infrared (NIR), and Sketch images. By reconstructing neutral facial prototypes and employing a feature-decoupling strategy, UMFN generates domain-agnostic features that preserve identity consistency. Experimental results on benchmarks like LAMP-HQ and CASIA NIR-VIS 2.0 demonstrate state-of-the-art performance in heterogeneous face recognition tasks.

Introduction

Standard face recognition systems often struggle with the "messiness" of real-world data, where subjects may exhibit non-frontal poses, varied expressions, or be captured in different spectral domains. Specifically, Heterogeneous Face Recognition (HFR) requires matching images across domains—such as comparing a near-infrared (NIR) night-vision frame or a forensic sketch to a high-quality visible-light (VIS) gallery photo. Existing methods typically specialize in a single type of variation (e.g., pose correction) or a specific domain pair, often failing when multiple degradations occur simultaneously or when the domain gap is substantial, leading to "identity inconsistency."

Methodology

The UMFN framework introduces a unified approach to face normalization by combining Autoencoders and Generative Adversarial Networks (GANs) to reconstruct a clean, neutral "facial prototype."

Feature Decoupling Mapping Network

The core of UMFN is a feature-decoupling strategy. Using a domain classifier and adversarial training, the system strips away domain-specific information (e.g., the stylistic markers of a sketch) from identity-relevant features. This process creates "domain-agnostic" representations that characterize the individual regardless of the capture modality.

Control Masks for Multi-Domain Generation

The decoder is guided by a 4-digit "control mask," allowing the network to perform both in-domain cleanup and cross-domain translation. For instance, an NIR input with a specific expression can be transformed into a normalized VIS prototype. This flexibility enables the model to bridge the gap between VIS, NIR, and Sketch domains within a single architecture.

Unified Multi-Domain Face Normalization results showing the transformation of various inputs (VIS pose, NIR expression, and Sketch) into normalized prototypes across all three domains.

Joint HFR Network with Contrastive Learning

To ensure identity preservation, UMFN employs a contrastive learning mechanism. The model aligns the features of a generated prototype with those of a real target photo in the feature space. By fusing domain-agnostic features from the original input with discriminative features from the generated prototype, the system maintains high recognition accuracy even if visual artifacts are present in the reconstruction.

Experimental Evaluation

The researchers evaluated UMFN on several diverse benchmarks:

  • LAMP-HQ: A large-scale NIR-VIS dataset (56,000+ images) featuring diverse poses and occlusions.
  • BUAA NIR-VIS & CASIA NIR-VIS 2.0: Standard benchmarks for infrared-to-visible matching.
  • CUFSF: A dataset for matching grayscale photos to sketches.
  • LFW & IJB-A: Classic "in-the-wild" datasets used to test generalization in unconstrained environments.

Results and Discussion

UMFN achieved state-of-the-art results across multiple HFR tasks. On the challenging LAMP-HQ dataset, the model reached a 99.1% Rank-1 accuracy, surpassing previous methods like NLPF (98.8%). Furthermore, when used as a pre-processing step for a standard ResNet-18 recognizer, it improved the Verification Rate (VR) on the IJB-A dataset by 5.35%.

Qualitative analysis indicates that UMFN produces more symmetrical and "clean" faces compared to baseline methods like DR-GAN or TP-GAN, particularly when handling heavy occlusions such as sunglasses. However, sketches remain a significant challenge due to their low information density, occasionally resulting in slight local distortions in the generated prototypes.

Conclusion

UMFN provides a robust, unified framework for face normalization that bridges the gap between multiple imaging domains while correcting for pose and expression. By focusing on "prototypes" rather than incremental fixes and utilizing feature decoupling, the model significantly enhances the performance of heterogeneous face recognition systems. Future work will focus on refining generation for low-detail inputs like sketches and extending the framework to additional modalities such as thermal imaging.

Summary updated

BibTeX
@inproceedings{pang2025umfn,
  author = {Meng Pang and Wenjun Zhang and Nanrun Zhou and Shengbo Chen and Hong Rao},
  title = {{UMFN: Unified Multi-Domain Face Normalization for Joint Cross-domain Prototype Learning and Heterogeneous Face Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02728},
}
Figure