Abstract
Facial landmark localization is a critical component of many computer vision tasks, yet it remains challenging under conditions of partial occlusion. This paper presents the Two-Stage Cascaded Pose Regression (TSCPR) framework, which leverages facial symmetry to improve localization robustness. By comparing predictions from original and mirrored images, the system identifies unreliable landmarks and applies corrections. A subsequent local stage employs a "divide and conquer" strategy to mitigate the impact of localized occlusions. Experimental results on the COFW dataset demonstrate that TSCPR achieves state-of-the-art performance in terms of accuracy and reliability.
Introduction
Identifying key facial landmarks—such as the corners of the eyes, nose tip, and mouth—is essential for applications ranging from facial recognition to emotion analysis. While existing algorithms perform well on clear, frontal images, their accuracy degrades significantly when faces are partially occluded by objects like sunglasses, hands, or hair. Standard Cascaded Pose Regression (CPR) methods often treat the face as a single global structure, which can be "broken" by partial occlusions. Furthermore, these models typically lack a mechanism to self-evaluate the reliability of their predictions.
Methods
The proposed TSCPR framework addresses these limitations through a two-stage process:
Stage 1: Global Regression and Correction
In the global stage, the system performs initial regression on both the original face image and its mirror reflection. By inverting the results of the mirror image and comparing them to the original, the model calculates a "mirror error" for each individual landmark. A high mirror error indicates that the model's prediction is inconsistent and likely unreliable due to occlusion. These landmarks are then adjusted using a Mirrorability-based Face Alignment (MFA) method.
Stage 2: Local Refinement
To further isolate the effects of occlusion, the global results are partitioned into four local regions: the left eye, right eye, nose, and mouth. Each region undergoes independent local regression. This approach ensures that an occlusion in one area (e.g., a hand over the mouth) does not negatively impact the localization accuracy in other, visible areas.
Key Insights: Mirroring as a Quality Proxy
A central innovation of this work is the use of mirror symmetry as a fine-grained quality check. The authors demonstrated a strong correlation (coefficient of ~0.81) between "mirror error" and actual "localization error." This validates mirror error as an effective proxy for accuracy in scenarios where ground truth data is unavailable during inference.
Experimental Evaluation
The TSCPR model was evaluated on the Caltech Occluded Faces in the Wild (COFW) dataset, a challenging benchmark characterized by heavy occlusions and diverse facial expressions.
Key Results
TSCPR outperformed several state-of-the-art models across multiple metrics:
- Accuracy: It achieved a Normalized Mean Error (NME) of 6.34, surpassing RICPR (6.64) and DRDA (6.46).
- Reliability: The failure rate (error > 0.1) was significantly reduced to 9.8%, compared to 20% for the baseline RCPR.
- Occlusion Detection: The model achieved an occlusion precision/recall of 80/57.1%, the highest among the compared methods.
Conclusion
The Two-Stage Cascaded Pose Regression framework provides a robust solution for facial landmark localization under occlusion. By integrating global mirror-consistency checks with local region-based refinement, TSCPR effectively identifies and corrects errors that traditional global models might miss. Future research could explore the integration of this two-stage logic into deep learning architectures and its performance in real-time video processing.
Summary updated