Large pose face recognition with morphing field learning

Lanqing Hu, Meina Kan, Shiguang Shan, Xilin ChenJournal of Image and Graphics 2022 · DOI

Abstract

Recognizing faces across large poses remains a significant challenge in computer vision. Traditional approaches typically rely on either 2D generative models, which produce natural-looking but identity-distorting results, or 3D morphable models, which preserve structure but lack flexibility. This paper presents a novel approach using a Coarse-to-Fine Morphing Field Network (CFMF-Net). By learning semantic-level pixel correspondences through a deep regression framework, the method achieves high-fidelity face frontalization. Evaluation across four major benchmarks—MultiPIE, LFW, CFP, and IJB-A—demonstrates that CFMF-Net outperforms existing state-of-the-art methods in recognition accuracy, particularly for extreme poses.

Introduction

Despite the rapid advancement of deep learning in facial recognition, "large pose" scenarios—where the subject is viewed at extreme angles (e.g., 75° to 90°)—continue to hinder performance. At these angles, critical identifying features are often occluded or severely distorted.

Current solutions generally follow two paradigms:

  1. 2D Generative Methods: Techniques like GANs synthesize frontal views. While visually appealing, they often introduce artifacts or "hallucinate" features that alter the subject's identity.
  2. 3D Modeling: These methods use physical models to rotate the face. While they maintain structural integrity, they are often rigid and struggle with non-planar deformations, leading to blurry or unnatural results in occluded regions.

CFMF-Net bridges this gap by combining the flexibility of 2D deep learning with the structural fidelity of 3D-like spatial transformations.

Methodology: Morphing Field Learning

The core innovation of this work is the Morphing Field. Unlike generative models that predict pixel colors, CFMF-Net predicts a 2D displacement vector for every pixel. This field defines a mapping from the target frontal image back to the original profile image, essentially "shifting" existing pixels into their correct frontal positions. This approach preserves original textures and identity-specific details that are often lost in synthesis.

Coarse-to-Fine Strategy

To ensure stable and accurate learning of the high-dimensional morphing field, the authors implement a multi-stage framework:

  • Landmark Extraction: The network first identifies 68 facial landmarks to establish the basic orientation and geometry of the face.
  • Coarse Morphing: A low-resolution field is generated to handle the global rotation and large-scale spatial shifts.
  • Detail Branch: A separate branch extracts fine-grained features from the input image to capture subtle textures.
  • Fine Morphing: The coarse field and fine features are fused to produce a dense, high-resolution morphing field that accounts for intricate facial details.

Symmetry and Pre-training

To handle occluded regions in profile views, the authors leverage facial symmetry as a prior. During pre-training, Thin Plate Spline (TPS) interpolation and symmetry-based estimation provide an initial morphing field, giving the network a robust starting point for end-to-end optimization.

Experimental Results

The CFMF-Net was rigorously tested against several benchmarks, showing consistent improvements over previous state-of-the-art (SOTA) models.

  • MultiPIE: In controlled environments with poses up to ±90°, CFMF-Net achieved an accuracy of 94.00%, surpassing the previous SOTA of 92.32%.
  • CFP (Celebrities in Frontal-Profile): On this challenging dataset designed for profile matching, the model reached 95.17% accuracy.
  • LFW and IJB-A: In "in-the-wild" scenarios involving unconstrained lighting and occlusions, the model maintained high performance, achieving a 95.3% true accept rate on IJB-A (at FAR=0.01).

Conclusion

The proposed Coarse-to-Fine Morphing Field Network successfully integrates the strengths of 2D and 3D face frontalization. By focusing on pixel-level semantic correspondence rather than synthesis, it preserves vital identity information while remaining flexible enough to handle complex deformations. Future work will explore sparsity-preserving structures to reduce the computational overhead of dense morphing fields and further integrate the frontalization process with downstream recognition tasks.

Summary updated

Abstract

目的人脸识别已经得到了广泛应用,但大姿态人脸识别问题仍未完美解决。已有方法或提取姿态鲁棒特征,或进行人脸姿态的正面化。其中主流的人脸正面化方法包括2D回归生成和3D模型形变建模,前者能够生成相对自然真实的人脸,但会引入额外的噪声导致图像信息的扭曲;后者能够保持原始的人脸结构信息,但生成过程是基于物理模型的,不够自然灵活。为此,结合2D和3D方法的优势,本文提出了基于由粗到细形变场的人脸正面化方法。方法该形变场由深度网络以2D回归方式学得,反映的是不同视角人脸图像像素之间的语义级对应关系,可以类3D的方式实现非正面人脸图像的正面化,因此该方法兼具了2D正面化方法的灵活性与3D正面化方法的保真性,且借鉴分步渐进的思路,本文提出了由粗到细的形变场学习框架,以获得更加准确鲁棒的形变场。结果本文采用大姿态人脸识别实验来验证本文方法的有效性,在MultiPIE (multi pose,illumination,expressions)、LFW (labeled faces in the wild)、CFP (celebrities in frontal-profile in the wild)、IJB-A (intelligence advanced research projects activity Janus benchmark-A)等4个数据集上均取得了比已有方法更高的人脸识别精度。结论本文提出的基于由粗到细的形变场学习的人脸正面化方法,综合了2D和3D人脸正面化方法的优点,使人脸正面化结果的学习更加灵活、准确,保持了更多有利于识别的身份信息。
BibTeX
@article{hu2022large,
  author = {Lanqing Hu and Meina Kan and Shiguang Shan and Xilin Chen},
  title = {{Large pose face recognition with morphing field learning}},
  journal = {Journal of Image and Graphics},
  year = {2022},
  doi = {10.11834/jig.210011},
}
Figure