Abstract
Facial keypoint detection is a critical component for applications ranging from facial recognition to driver monitoring. While Deep Convolutional Neural Networks (DCNNs) excel at feature extraction, they often struggle to model the complex geometric relationships between facial landmarks, particularly under extreme poses or occlusions. This paper proposes a novel single-shot architecture that utilizes a "convolution tree" with "deconvolution branches" to explicitly model these spatial dependencies. By incorporating a pose-based routing function, the network dynamically adjusts information flow between keypoints based on the estimated 3D head orientation. Our method achieves state-of-the-art performance on challenging benchmarks like AFLW, demonstrating significant robustness to pose variation while maintaining efficient processing speeds.
Introduction
Facial keypoint detection, or face alignment, involves identifying specific landmarks such as the eyes, nose, and mouth. Despite the success of DCNNs, capturing the underlying geometric constraints—such as the relative position of an eye to the nose tip—remains a challenge. Traditional "cascade regression" methods address this through iterative refinement, but these are often computationally expensive and prone to failure in profile views or under partial occlusion. The core difficulty lies in modeling these geometric relationships in a way that accounts for how they change with the head's 3D pose.
Methodology
The proposed architecture avoids iterative refinement by using a multi-task, single-shot approach.
Convolution-Deconvolution Architecture
The network extracts a shared "image code" using a Squeezenet-based backbone. This code is then processed through multiple deconvolutional branches, each dedicated to a specific facial keypoint. These branches generate response maps (heatmaps) that represent the probability distribution of each landmark's location.
Message Passing and the Tree Structure
To capture geometric constraints, the model implements a message-passing mechanism between keypoint branches. Instead of a dense, computationally heavy all-to-all connection, keypoints are organized into a tree structure. The "nose tip" serves as the root, passing learned geometric transformations to adjacent features. This allows the model to infer the location of one feature based on the positions of others.
Pose-Based Routing
A key innovation is the pose-based routing function. Since geometric relationships vary with head orientation, the network uses the predicted 3D head pose (yaw, pitch, and roll) to modulate the flow of information through the deconvolutional branches. This dynamic routing allows the model to "hallucinate" occluded points by leveraging visible landmarks and the overall head pose.
Evaluation and Results
The model was evaluated on the AFLW (Annotated Facial Landmarks in the Wild) and AFW (Annotated Faces in the Wild) datasets, both known for extreme pose variations.
Accuracy and Robustness
On the AFLW-PIFA protocol, the model achieved a Normalized Mean Error (NME) of 3.93, a substantial improvement over previous methods like PIFA (6.8) and 3DDFA (5.32). Notably, the model maintained high accuracy even at extreme yaw angles (60°-90°), where traditional methods typically fail.
Efficiency
By utilizing lightweight Squeezenet modules for the deconvolution branches, the architecture remains efficient. It achieves a processing speed of 7-8 frames per second (FPS) on a single GPU, outperforming many deep cascade-based architectures.
Conclusion
This work demonstrates that incorporating explicit geometric modeling and pose-dependent routing into a single-shot DCNN significantly enhances facial keypoint detection. The "convolution tree" approach effectively handles occlusions and extreme poses by sharing information across landmarks. Future work may focus on further optimizing the backbone for real-time mobile applications and extending this tree-based message-passing framework to other tasks, such as human body pose estimation.
Summary updated