Abstract
Recognizing faces across large poses remains a significant challenge in computer vision. Traditional approaches typically rely on either 2D generative models, which produce natural-looking but identity-distorting results, or 3D morphable models, which preserve structure but lack flexibility. This paper presents a novel approach using a Coarse-to-Fine Morphing Field Network (CFMF-Net). By learning semantic-level pixel correspondences through a deep regression framework, the method achieves high-fidelity face frontalization. Evaluation across four major benchmarks—MultiPIE, LFW, CFP, and IJB-A—demonstrates that CFMF-Net outperforms existing state-of-the-art methods in recognition accuracy, particularly for extreme poses.
Introduction
Despite the rapid advancement of deep learning in facial recognition, "large pose" scenarios—where the subject is viewed at extreme angles (e.g., 75° to 90°)—continue to hinder performance. At these angles, critical identifying features are often occluded or severely distorted.
Current solutions generally follow two paradigms:
- 2D Generative Methods: Techniques like GANs synthesize frontal views. While visually appealing, they often introduce artifacts or "hallucinate" features that alter the subject's identity.
- 3D Modeling: These methods use physical models to rotate the face. While they maintain structural integrity, they are often rigid and struggle with non-planar deformations, leading to blurry or unnatural results in occluded regions.
CFMF-Net bridges this gap by combining the flexibility of 2D deep learning with the structural fidelity of 3D-like spatial transformations.
Methodology: Morphing Field Learning
The core innovation of this work is the Morphing Field. Unlike generative models that predict pixel colors, CFMF-Net predicts a 2D displacement vector for every pixel. This field defines a mapping from the target frontal image back to the original profile image, essentially "shifting" existing pixels into their correct frontal positions. This approach preserves original textures and identity-specific details that are often lost in synthesis.
Coarse-to-Fine Strategy
To ensure stable and accurate learning of the high-dimensional morphing field, the authors implement a multi-stage framework:
- Landmark Extraction: The network first identifies 68 facial landmarks to establish the basic orientation and geometry of the face.
- Coarse Morphing: A low-resolution field is generated to handle the global rotation and large-scale spatial shifts.
- Detail Branch: A separate branch extracts fine-grained features from the input image to capture subtle textures.
- Fine Morphing: The coarse field and fine features are fused to produce a dense, high-resolution morphing field that accounts for intricate facial details.
Symmetry and Pre-training
To handle occluded regions in profile views, the authors leverage facial symmetry as a prior. During pre-training, Thin Plate Spline (TPS) interpolation and symmetry-based estimation provide an initial morphing field, giving the network a robust starting point for end-to-end optimization.
Experimental Results
The CFMF-Net was rigorously tested against several benchmarks, showing consistent improvements over previous state-of-the-art (SOTA) models.
- MultiPIE: In controlled environments with poses up to ±90°, CFMF-Net achieved an accuracy of 94.00%, surpassing the previous SOTA of 92.32%.
- CFP (Celebrities in Frontal-Profile): On this challenging dataset designed for profile matching, the model reached 95.17% accuracy.
- LFW and IJB-A: In "in-the-wild" scenarios involving unconstrained lighting and occlusions, the model maintained high performance, achieving a 95.3% true accept rate on IJB-A (at FAR=0.01).
Conclusion
The proposed Coarse-to-Fine Morphing Field Network successfully integrates the strengths of 2D and 3D face frontalization. By focusing on pixel-level semantic correspondence rather than synthesis, it preserves vital identity information while remaining flexible enough to handle complex deformations. Future work will explore sparsity-preserving structures to reduce the computational overhead of dense morphing fields and further integrate the frontalization process with downstream recognition tasks.
Summary updated