A Convolution Tree with Deconvolution Branches: Exploiting Geometric Relationships for Single Shot Keypoint Detection

Amit Kumar, Rama ChellappaarXiv 2017 · arXiv

Abstract

Facial keypoint detection is a critical component for applications ranging from facial recognition to driver monitoring. While Deep Convolutional Neural Networks (DCNNs) excel at feature extraction, they often struggle to model the complex geometric relationships between facial landmarks, particularly under extreme poses or occlusions. This paper proposes a novel single-shot architecture that utilizes a "convolution tree" with "deconvolution branches" to explicitly model these spatial dependencies. By incorporating a pose-based routing function, the network dynamically adjusts information flow between keypoints based on the estimated 3D head orientation. Our method achieves state-of-the-art performance on challenging benchmarks like AFLW, demonstrating significant robustness to pose variation while maintaining efficient processing speeds.

Introduction

Facial keypoint detection, or face alignment, involves identifying specific landmarks such as the eyes, nose, and mouth. Despite the success of DCNNs, capturing the underlying geometric constraints—such as the relative position of an eye to the nose tip—remains a challenge. Traditional "cascade regression" methods address this through iterative refinement, but these are often computationally expensive and prone to failure in profile views or under partial occlusion. The core difficulty lies in modeling these geometric relationships in a way that accounts for how they change with the head's 3D pose.

Methodology

The proposed architecture avoids iterative refinement by using a multi-task, single-shot approach.

Convolution-Deconvolution Architecture

The network extracts a shared "image code" using a Squeezenet-based backbone. This code is then processed through multiple deconvolutional branches, each dedicated to a specific facial keypoint. These branches generate response maps (heatmaps) that represent the probability distribution of each landmark's location.

Message Passing and the Tree Structure

To capture geometric constraints, the model implements a message-passing mechanism between keypoint branches. Instead of a dense, computationally heavy all-to-all connection, keypoints are organized into a tree structure. The "nose tip" serves as the root, passing learned geometric transformations to adjacent features. This allows the model to infer the location of one feature based on the positions of others.

Pose-Based Routing

A key innovation is the pose-based routing function. Since geometric relationships vary with head orientation, the network uses the predicted 3D head pose (yaw, pitch, and roll) to modulate the flow of information through the deconvolutional branches. This dynamic routing allows the model to "hallucinate" occluded points by leveraging visible landmarks and the overall head pose.

Visualization of the model's output, showing the input face, generated heatmaps, and final keypoint detection with estimated 3D pose.

Evaluation and Results

The model was evaluated on the AFLW (Annotated Facial Landmarks in the Wild) and AFW (Annotated Faces in the Wild) datasets, both known for extreme pose variations.

Accuracy and Robustness

On the AFLW-PIFA protocol, the model achieved a Normalized Mean Error (NME) of 3.93, a substantial improvement over previous methods like PIFA (6.8) and 3DDFA (5.32). Notably, the model maintained high accuracy even at extreme yaw angles (60°-90°), where traditional methods typically fail.

Efficiency

By utilizing lightweight Squeezenet modules for the deconvolution branches, the architecture remains efficient. It achieves a processing speed of 7-8 frames per second (FPS) on a single GPU, outperforming many deep cascade-based architectures.

Conclusion

This work demonstrates that incorporating explicit geometric modeling and pose-dependent routing into a single-shot DCNN significantly enhances facial keypoint detection. The "convolution tree" approach effectively handles occlusions and extreme poses by sharing information across landmarks. Future work may focus on further optimizing the backbone for real-time mobile applications and extending this tree-based message-passing framework to other tasks, such as human body pose estimation.

Summary updated

Abstract

Recently, Deep Convolution Networks (DCNNs) have been applied to the task of face alignment and have shown potential for learning improved feature representations. Although deeper layers can capture abstract concepts like pose, it is difficult to capture the geometric relationships among the keypoints in DCNNs. In this paper, we propose a novel convolution-deconvolution network for facial keypoint detection. Our model predicts the 2D locations of the keypoints and their individual visibility along with 3D head pose, while exploiting the spatial relationships among different keypoints. Different from existing approaches of modeling these relationships, we propose learnable transform functions which captures the relationships between keypoints at feature level. However, due to extensive variations in pose, not all of these relationships act at once, and hence we propose, a pose-based routing function which implicitly models the active relationships. Both transform functions and the routing function are implemented through convolutions in a multi-task framework. Our approach presents a single-shot keypoint detection method, making it different from many existing cascade regression-based methods. We also show that learning these relationships significantly improve the accuracy of keypoint detections for in-the-wild face images from challenging datasets such as AFW and AFLW.
BibTeX
@misc{kumar2017convolution,
  author = {Amit Kumar and Rama Chellappa},
  title = {{A Convolution Tree with Deconvolution Branches: Exploiting Geometric Relationships for Single Shot Keypoint Detection}},
  year = {2017},
  eprint = {1704.01880},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1704.01880},
}
Figure