Face Alignment With Kernel Density Deep Neural Network

Lisha Chen, Hui Su, Qiang JiICCV 2019 · OpenReview

Abstract

Deep neural networks achieve good performance in many computer vision problems such as face alignment. However, when the testing image is challenging due to low resolution, occlusion or adversarial attacks, the accuracy of a deep neural network suffers greatly. Therefore, it is important to quantify the uncertainty in its predictions. A probabilistic neural network with Gaussian distribution over the target is typically used to quantify uncertainty for regression problems. However, in real-world problems especially computer vision tasks, the Gaussian assumption is too strong. To model more general distributions, such as multi-modal or asymmetric distributions, we propose to develop a kernel density deep neural network. Specifically, for face alignment, we adapt state-of-the-art hourglass neural network into a probabilistic neural network framework with landmark probability map as its output. The model is trained by maximizing the conditional log likelihood. To exploit the output probability map, we extend the model to multi-stage so that the logits map from the previous stage can feed into the next stage to progressively improve the landmark detection accuracy. Extensive experiments on benchmark datasets against state-of-the-art unconstrained deep learning method demonstrate that the proposed kernel density network achieves comparable or superior performance in terms of prediction accuracy. It further provides aleatoric uncertainty estimation in predictions.

Introduction

Face alignment—the task of identifying specific facial landmarks like the corners of eyes or the tip of the nose—is a foundational problem in computer vision. While current deep learning models perform well on clear, frontal images, their accuracy degrades significantly under challenging conditions such as low resolution, heavy occlusion, or adversarial attacks.

Most existing methods rely on "heatmap regression," where a network predicts a fixed Gaussian "blob" around a landmark. This approach has two major drawbacks: it assumes a symmetric Gaussian error distribution, which is often violated in real-world scenarios (e.g., asymmetric probability due to occlusion), and it fails to provide a measure of prediction uncertainty. This paper introduces the Kernel Density Deep Neural Network (KDN) to overcome these limitations by modeling landmark locations as continuous probability distributions.

Methods

The core innovation of this work is the transition from point-based regression to continuous probability modeling using kernel density estimation.

Kernel Density Modeling

Instead of a fixed heatmap, the KDN outputs a weight map that defines a mixture of Gaussian kernels. This allows the model to represent a continuous probability surface $p(y|x)$ that can adapt its shape—becoming multimodal or skewed—based on the input image. This flexibility is essential for handling ambiguity; for instance, if an eye is occluded, the model can represent multiple likely locations rather than forcing a single, potentially incorrect, high-confidence prediction.

Multi-Stage Cascade Architecture

As illustrated in Figure 1, the proposed architecture utilizes a multi-stage cascade of "Stacked Hourglass" networks. The logits map from one stage is passed to the next, allowing for progressive refinement. If an early stage is uncertain (indicated by a broad, flat probability map), the subsequent stage searches a wider area. Conversely, a sharp peak in the early stages allows later stages to focus on fine-tuning the landmark's coordinates.

Architecture of the multi-stage Kernel Density Network (KDN) showing the flow from input face to final probability maps.

Likelihood-Based Training and Continuous Inference

KDN is trained by maximizing the conditional log-likelihood, a more mathematically rigorous objective for probabilistic models than standard Mean Squared Error (MSE). During inference, the model performs gradient ascent on the predicted distribution. This allows it to find the most likely landmark location in continuous space, bypassing the resolution limits of traditional output grids and achieving sub-pixel accuracy.

Experimental Results

The KDN was evaluated on several rigorous benchmarks, including 300W, Menpo, AFLW, and the occlusion-focused COFW dataset.

Accuracy and Robustness

The KDN-Gaussian model achieved a Normalized Mean Error (NME) of 2.21% on the 300W test set, surpassing state-of-the-art models like FAN and SAN. As shown in the Cumulative Error Distribution (CED) curves in Figure 2, the KDN variants consistently outperform competing methods across various error thresholds.

Cumulative Error Distribution (CED) curves comparing KDN variants against state-of-the-art methods on benchmark datasets.

Uncertainty as a Signal

A key advantage of the KDN is its ability to quantify aleatoric uncertainty. The variance or "flatness" of the predicted probability map serves as an effective built-in occlusion detector. Experiments on the COFW-68 dataset showed that KDN's negative log-likelihood (NLL) scores were significantly better than standard methods (3.21 vs. 5.27), indicating that its probabilistic predictions are much better aligned with the actual uncertainty of landmark locations.

Conclusion

The Kernel Density Network provides a robust framework for face alignment by modeling landmark locations as flexible, continuous probability distributions. While the multi-stage cascade and gradient ascent inference introduce some computational overhead (running at ~10 fps), the gains in accuracy and the ability to quantify uncertainty represent a significant step forward for robust computer vision. Future work will explore extending this kernel density approach to other regression tasks such as human pose estimation.

Summary updated

Abstract

Deep neural networks achieve good performance in many computer vision problems such as face alignment. However, when the testing image is challenging due to low resolution, occlusion or adversarial attacks, the accuracy of a deep neural network suffers greatly. Therefore, it is important to quantify the uncertainty in its predictions. A probabilistic neural network with Gaussian distribution over the target is typically used to quantify uncertainty for regression problems. However, in real-world problems especially computer vision tasks, the Gaussian assumption is too strong. To model more general distributions, such as multi-modal or asymmetric distributions, we propose to develop a kernel density deep neural network. Specifically, for face alignment, we adapt state-of-the-art hourglass neural network into a probabilistic neural network framework with landmark probability map as its output. The model is trained by maximizing the conditional log likelihood. To exploit the output probability map, we extend the model to multi-stage so that the logits map from the previous stage can feed into the next stage to progressively improve the landmark detection accuracy. Extensive experiments on benchmark datasets against state-of-the-art unconstrained deep learning method demonstrate that the proposed kernel density network achieves comparable or superior performance in terms of prediction accuracy. It further provides aleatoric uncertainty estimation in predictions.
BibTeX
@inproceedings{chen2019face,
  author = {Lisha Chen and Hui Su and Qiang Ji},
  title = {{Face Alignment With Kernel Density Deep Neural Network}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=nZs1NtOIu1m},
}
Figure