Robust facial landmark detection by cross-order cross-semantic deep network

Jun Wan, Zhihui Lai, Linlin Shen, Jie Zhou, Can Gao, Gang Xiao, Xianxu Hou

TL;DR

Facial landmark detection often fails when faces are heavily occluded or turned at extreme angles. This paper introduces the Cross-order Cross-semantic Deep Network (CCDN), an architecture that uses both first- and second-order statistics to activate specific facial regions, explicitly forcing the network to learn complementary features. This "divide and conquer" approach significantly boosts accuracy and robustness in challenging, real-world scenarios.

What Problem Does This Paper Address?

Facial landmark detection—the task of mapping key points like the corners of the eyes, tip of the nose, and edges of the mouth—is a foundational step for face recognition, emotion analysis, and augmented reality filters. While modern Convolutional Neural Networks (CNNs) are great at this on clear, front-facing images, they struggle in "the wild."

When a face is turned sharply (large pose variation) or partially covered by hands, hair, or objects (heavy occlusion), standard CNNs falter. Existing methods typically regress coordinates from the whole face at once or generate heatmaps for individual points. However, they fail to explicitly model the relationships between different facial parts. If the mouth is covered, the network doesn't effectively use the visible eyes and nose to infer the mouth's location. The authors argue that networks need to learn "part-specific semantic features"—essentially training different parts of the network to become experts on different facial regions—to survive extreme occlusions and poses.

Methods

To solve this, the authors propose the Cross-order Cross-semantic Deep Network (CCDN). It builds on a standard stacked hourglass network backbone but introduces two major novel components:

  1. Cross-order Two-squeeze Multi-excitation (CTM) Module: Standard attention mechanisms (like SENet) use "first-order" statistics (Global Average Pooling) to figure out which feature channels are most important. The CTM module goes a step further by also using "second-order" statistics (Global Covariance Pooling). By looking at the variance and relationships between channels, the network can make much smarter decisions about which features to emphasize. Furthermore, instead of outputting one attention map, the CTM outputs multiple attention-specific feature maps.

  2. Cross-order Cross-semantic (COCS) Regularizer: If you have multiple attention maps, how do you ensure they don't all just look at the easiest feature (like the nose)? The COCS regularizer is a custom loss function that forces these different attention blocks to learn different semantic meanings. It maximizes the correlation of features within the same block while minimizing the correlation between different blocks. Finally, a Cross-layer Feature Fusing (CFF) block aggregates these diverse features across different depths of the network.

Datasets

The authors evaluated their model on four highly challenging benchmark datasets:

  • 300W: A standard competition dataset, specifically focusing on its "Challenging Subset" (IBUG) which features faces in the wild.
  • COFW-68 (Caltech Occluded Faces in the Wild): A dataset specifically designed to test occlusion, where faces are heavily covered by objects or hands.
  • AFLW: Contains faces with extreme pose variations (yaw angles from -120° to +120°).
  • WFLW (Wider Facial Landmarks in-the-wild): A rich dataset with 98 landmarks and specific attribute annotations (occlusion, pose, make-up, illumination, blur, expression).

Key Insights and Ideas

  • Second-order statistics unlock better attention: Relying only on average pooling leaves valuable data on the table. By incorporating covariance (second-order statistics), the network gains a much more discriminative understanding of how different feature channels interact, leading to better recalibration of important features.
  • Forced complementarity builds resilience: The COCS regularizer is the "aha" moment of the paper. By mathematically forcing different attention heads to decorrelate, the network naturally learns to assign different heads to different facial parts (e.g., one head tracks the jawline, another tracks the eyes). If a hand covers the mouth, the eye-tracking and jaw-tracking heads are unaffected, keeping the overall facial structure intact.
  • Visualizing the "Black Box": The authors generated face activation maps showing that their multiple excitation blocks literally highlight different, complementary regions of the face (e.g., one block highlights the forehead/eyes, another highlights the lower face).

Key Results

The CCDN consistently outperformed state-of-the-art methods across the board, particularly in edge-case scenarios:

  • Heavy Occlusion (COFW): Achieved a Normalized Mean Error (NME) of 4.91% and dropped the failure rate to an impressive 1.12%.
  • Extreme Poses (AFLW): Reached an NME of 1.35% on the full testing set.
  • Complex Attributes (WFLW): Achieved an NME of 4.29% on the test set, beating competitors across all specific subsets (pose, expression, illumination, makeup, occlusion, and blur).
  • Ablation: Removing the second-order statistics or the COCS regularizer noticeably degraded performance, proving that both novel additions are necessary for the accuracy gains.

Limitations and Future Work

The primary limitation acknowledged by the authors is computational cost. The robustness of the CCDN relies on having multiple "excitation blocks" (attention heads) to cover different facial parts. The authors found that increasing the number of blocks increases accuracy, but it also linearly increases the computational load. In their experiments, they settled on using 4 excitation blocks as the optimal sweet spot to balance high accuracy with reasonable computational cost. Future work could likely explore ways to make this multi-head attention process more computationally efficient for real-time, edge-device applications.

Summary updated

Abstract

Recently, convolutional neural networks (CNNs)-based facial landmark detection methods have achieved great success. However, most of existing CNN-based facial landmark detection methods have not attempted to activate multiple correlated facial parts and learn different semantic features from them that they can not accurately model the relationships among the local details and can not fully explore more discriminative and fine semantic features, thus they suffer from partial occlusions and large pose variations. To address these problems, we propose a cross-order cross-semantic deep network (CCDN) to boost the semantic features learning for robust facial landmark detection. Specifically, a cross-order two-squeeze multi-excitation (CTM) module is proposed to introduce the cross-order channel correlations for more discriminative representations learning and multiple attention-specific part activation. Moreover, a novel cross-order cross-semantic (COCS) regularizer is designed to drive the network to learn cross-order cross-semantic features from different activation for facial landmark detection. It is interesting to show that by integrating the CTM module and COCS regularizer, the proposed CCDN can effectively activate and learn more fine and complementary cross-order cross-semantic features to improve the accuracy of facial landmark detection under extremely challenging scenarios. Experimental results on challenging benchmark datasets demonstrate the superiority of our CCDN over state-of-the-art facial landmark detection methods.
BibTeX
@inproceedings{wan2021robust,
  author = {Jun Wan and Zhihui Lai and Linlin Shen and Jie Zhou and Can Gao and Gang Xiao and Xianxu Hou},
  title = {{Robust facial landmark detection by cross-order cross-semantic deep network}},
  booktitle = {Neural Networks},
  year = {2021},
  url = {https://openreview.net/forum?id=QKP1z0iwV2N},
}

Loading related content...

Figure