Robust Facial Landmark Localization Based on Two-Stage Cascaded Pose Regression

Ziye Tong, Junwei Zhou, Yanchao Yang, Lee-Ming ChengAAAI 2019 · DOI

Abstract

Facial landmark localization is a critical component of many computer vision tasks, yet it remains challenging under conditions of partial occlusion. This paper presents the Two-Stage Cascaded Pose Regression (TSCPR) framework, which leverages facial symmetry to improve localization robustness. By comparing predictions from original and mirrored images, the system identifies unreliable landmarks and applies corrections. A subsequent local stage employs a "divide and conquer" strategy to mitigate the impact of localized occlusions. Experimental results on the COFW dataset demonstrate that TSCPR achieves state-of-the-art performance in terms of accuracy and reliability.

Introduction

Identifying key facial landmarks—such as the corners of the eyes, nose tip, and mouth—is essential for applications ranging from facial recognition to emotion analysis. While existing algorithms perform well on clear, frontal images, their accuracy degrades significantly when faces are partially occluded by objects like sunglasses, hands, or hair. Standard Cascaded Pose Regression (CPR) methods often treat the face as a single global structure, which can be "broken" by partial occlusions. Furthermore, these models typically lack a mechanism to self-evaluate the reliability of their predictions.

Methods

The proposed TSCPR framework addresses these limitations through a two-stage process:

Stage 1: Global Regression and Correction

In the global stage, the system performs initial regression on both the original face image and its mirror reflection. By inverting the results of the mirror image and comparing them to the original, the model calculates a "mirror error" for each individual landmark. A high mirror error indicates that the model's prediction is inconsistent and likely unreliable due to occlusion. These landmarks are then adjusted using a Mirrorability-based Face Alignment (MFA) method.

Stage 2: Local Refinement

To further isolate the effects of occlusion, the global results are partitioned into four local regions: the left eye, right eye, nose, and mouth. Each region undergoes independent local regression. This approach ensures that an occlusion in one area (e.g., a hand over the mouth) does not negatively impact the localization accuracy in other, visible areas.

The TSCPR framework architecture, illustrating the Global Stage (I) and Local Stage (II) for robust landmark localization.

Key Insights: Mirroring as a Quality Proxy

A central innovation of this work is the use of mirror symmetry as a fine-grained quality check. The authors demonstrated a strong correlation (coefficient of ~0.81) between "mirror error" and actual "localization error." This validates mirror error as an effective proxy for accuracy in scenarios where ground truth data is unavailable during inference.

Experimental Evaluation

The TSCPR model was evaluated on the Caltech Occluded Faces in the Wild (COFW) dataset, a challenging benchmark characterized by heavy occlusions and diverse facial expressions.

Key Results

TSCPR outperformed several state-of-the-art models across multiple metrics:

  • Accuracy: It achieved a Normalized Mean Error (NME) of 6.34, surpassing RICPR (6.64) and DRDA (6.46).
  • Reliability: The failure rate (error > 0.1) was significantly reduced to 9.8%, compared to 20% for the baseline RCPR.
  • Occlusion Detection: The model achieved an occlusion precision/recall of 80/57.1%, the highest among the compared methods.
Comparison of TSCPR with state-of-the-art methods on the COFW dataset, showing improvements in NME, failure rate, and occlusion detection.

Conclusion

The Two-Stage Cascaded Pose Regression framework provides a robust solution for facial landmark localization under occlusion. By integrating global mirror-consistency checks with local region-based refinement, TSCPR effectively identifies and corrects errors that traditional global models might miss. Future research could explore the integration of this two-stage logic into deep learning architectures and its performance in real-time video processing.

Summary updated

BibTeX
@inproceedings{tong2019robust,
  author = {Ziye Tong and Junwei Zhou and Yanchao Yang and Lee-Ming Cheng},
  title = {{Robust Facial Landmark Localization Based on Two-Stage Cascaded Pose Regression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.330110055},
}
Figure