Learning to validate the quality of detected landmarks

Wolfgang Fuhl, Enkelejda Kasneci

TL;DR

This paper introduces a novel approach to improve facial landmark detection by simultaneously learning landmark positions and their reliability (validation signal). They also propose a "Loss Probability based Batch Balancing" (LPBB) method to dynamically prioritize challenging training samples, leading to more robust and accurate landmark detection.

What Problem Does This Paper Address?

Accurate facial landmark detection is crucial for many computer vision tasks like face alignment, reconstruction, and recognition. While deep learning models have achieved impressive results, they can still struggle with variations in pose, illumination, and occlusion, leading to unreliable landmark predictions. Existing methods often treat all detected landmarks equally, even those with low confidence. This paper tackles two key challenges: 1) how to make a model aware of its own prediction quality, and 2) how to effectively train models on datasets with imbalanced challenges (e.g., certain poses or occlusions being underrepresented).

Methods

The core innovation lies in a new loss function that extends traditional regression losses. Instead of just predicting the landmark's (x, y) coordinates, the network also learns a "validation signal." This signal is designed to estimate the distance between the predicted and ground truth landmark position, essentially acting as a confidence score. The authors formulate this by adding an extra output neuron per landmark that learns to predict the Manhattan distance between the predicted and ground truth coordinates.

Complementing this, they introduce a "Loss Probability based Batch Balancing" (LPBB) method. Instead of uniform sampling of training data, LPBB dynamically selects samples for each training batch based on their computed loss from the previous iteration. Samples with higher losses (indicating they were harder to predict) are given a higher probability of being included in the next batch. This ensures that challenging examples, regardless of their specific pose or occlusion, are more frequently seen by the network, leading to better generalization.

Datasets

The authors evaluated their methods on three widely used facial landmark datasets:

  • 300W: A comprehensive dataset aggregating multiple sources, with different subsets like "Common," "Challenging," and "Full."
  • AFLW: Known for its wide range of pose variations.
  • WFLW: Includes various challenges like blur, makeup, occlusion, and expression.

Key Insights and Ideas

  • Self-Awareness in Landmark Detection: By training the network to predict its own error, we can gain a measure of confidence for each detected landmark. This "validation signal" can then be used to filter out unreliable predictions.
  • Dynamic Training Data Prioritization: LPBB is a powerful way to address data imbalance beyond just pose. It automatically focuses training on samples that are currently difficult for the model, leading to more robust learning.
  • Correlation is Key: The validation signal learned by the network shows a strong correlation with the actual landmark error across all tested datasets, validating its effectiveness.
  • Beyond Pose Balancing: While pose balancing is important, LPBB addresses a broader range of challenges (occlusion, illumination, etc.) by using the loss as a proxy for difficulty.

Key Results

  • The proposed validation loss, when used with batch balancing (models BBO and BB), significantly improves landmark detection accuracy across the 300W and AFLW datasets compared to models without validation loss or batch balancing (NON and NOBB).
  • For instance, on the 300W dataset, the "BB" model (validation loss + batch balancing) achieved a normalized mean error (NME) of 3.69 for the "Full" set, outperforming the "NON" model's 4.35.
  • The "BB" model also showed strong performance on challenging subsets, achieving 4.66 on 300W "Challenging," compared to 7.38 for "NOBB."
  • On the WFLW dataset, the batch balancing approach (BB) was particularly effective for challenges like blur, occlusion, and pose, outperforming even state-of-the-art methods in some cases.
  • The validation signal can be used to discard a percentage of landmarks (e.g., 30%) to further improve accuracy, with the "BB (Discard 30%)" model achieving an NME of 2.74 on 300W Full.
  • In terms of speed, the proposed architecture achieves competitive frame rates (e.g., 333 fps) while delivering superior accuracy.

Limitations and Future Work

  • Competition Between Tasks: The paper notes that the validation signal can sometimes compete with the landmark position prediction, potentially reducing accuracy in some scenarios when only the validation loss is used without batch balancing.
  • Computational Cost of LPBB: While LPBB is effective, it requires evaluating all data after each batch to compute losses, which can be computationally intensive. The paper mentions exploring two different uses for evaluation, suggesting a trade-off between accuracy and computational effort.
  • Future Directions: The authors suggest that the validation signal could be used for secondary validation of face detection and for handling other imbalances in the data beyond just head pose. They also mention the potential for using this signal to rule out inaccurate landmarks in subsequent processing steps.

Summary updated

Abstract

We present a new loss function for the validation of image landmarks detected via Convolutional Neural Networks (CNN). The network learns to estimate how accurate its landmark estimation is. This loss function is applicable to all regression-based location estimations and allows the exclusion of unreliable landmarks from further processing. In addition, we formulate a novel batch balancing approach which weights the importance of samples based on their produced loss. This is done by computing a probability distribution mapping on an interval from which samples can be selected using a uniform random selection scheme. We conducted experiments on the 300W, AFLW, and WFLW facial landmark datasets. In the first experiments, the influence of our batch balancing approach is evaluated by comparing it against uniform sampling. In addition, we evaluated the impact of the validation loss on the landmark accuracy based on uniform sampling. The last experiments evaluate the correlation of the validation signal with the landmark accuracy. All experiments were performed for all three datasets.
BibTeX
@inproceedings{fuhl2019learning,
  author = {Wolfgang Fuhl and Enkelejda Kasneci},
  title = {{Learning to validate the quality of detected landmarks}},
  booktitle = {ICMV},
  year = {2019},
  url = {https://openreview.net/forum?id=xhXuY-viuUh},
}
Figure