Robust Facial Landmark Detection via Aggregation on Geometrically Manipulated Faces

Seyed Mehdi Iranmanesh, Ali Dabouei, Sobhan Soleymani, Hadi Kazemi, Nasser NasrabadiWACV 2020

TL;DR

This paper introduces GEAN, a novel framework for robust facial landmark detection that tackles challenges posed by significant shape and appearance variations. GEAN achieves this by aggregating predictions from multiple geometrically manipulated versions of an input face image, making it more resilient to distortions.

What Problem Does This Paper Address?

Accurately detecting facial landmarks (key points like the tip of the nose or corners of the eyes) is crucial for many computer vision tasks, including 3D face reconstruction and recognition. However, existing methods struggle with the inherent variability in faces due to factors like expression changes, head pose, and occlusions. While deep learning models have shown promise, they can be surprisingly sensitive to small perturbations in input images, leading to drastic changes in landmark predictions. This paper aims to create a landmark detection system that is inherently robust to these variations by leveraging geometric manipulations.

Methods

The core of the proposed method, GEAN (Geometry Aggregated Network), is an aggregation framework. It works by:

  1. Generating Manipulated Faces: For a given input face image, the framework generates K different manipulated versions. These manipulations are designed to introduce geometric perturbations that can fool face recognition models. Three approaches are proposed for this:
  2. Adversarial Attacks: Using techniques inspired by adversarial attacks to subtly alter the image, aiming to change its identity representation while preserving visual similarity.
  3. Semantic Grouping with Adversarial Attacks: A refinement of the adversarial approach where landmarks are grouped semantically (e.g., all landmarks related to the left eye) and manipulated together. This helps preserve the overall facial structure.
  4. Semantic Grouping with Known Transformations: Using predefined geometric transformations (like scaling and translation) applied to semantically grouped landmarks.
  5. Shared Landmark Detector: A single, powerful landmark detector (an hourglass network) is used to predict landmarks for each of the K manipulated faces.
  6. Inverse Transformation and Aggregation: The predicted landmarks from the manipulated faces are transformed back to their original coordinate space using the inverse of the manipulation transformation. These transformed landmarks are then aggregated, weighted by a score that reflects the displacement caused by the manipulation. This weighted aggregation produces the final, robust landmark prediction.

Datasets

The proposed method was evaluated on the following standard facial landmark detection datasets:

  • 300-W: A widely used benchmark dataset that combines several existing datasets (LFPW, HELEN, AFW, IBUG, XM2VTS) with a total of 68 landmarks.
  • AFLW: A large-scale dataset with a wide variety of poses, expressions, and environmental conditions, providing up to 21 landmarks per face.
  • COFW: A dataset with 68 landmarks, originally annotated with fewer points.

Key Insights and Ideas

  • Sensitivity of Deep Models: The paper highlights a critical vulnerability of deep learning models: their sensitivity to small input perturbations, which can lead to significant errors in downstream tasks like landmark detection.
  • Leveraging Manipulations for Robustness: Instead of solely relying on robust training data, the paper proposes actively creating challenging, manipulated versions of the input to force the model to learn more invariant features.
  • Semantic Grouping is Key: Manipulating landmarks semantically (grouping related points like those of an eye) is shown to be more effective than independent landmark manipulation, as it preserves the overall facial structure and avoids severe distortions.
  • Aggregation as a Solution: Combining predictions from multiple manipulated views through a weighted aggregation process significantly improves robustness and accuracy.

Key Results

  • The proposed GEAN method, particularly the GEANGadv variant (semantic grouping with adversarial attacks), consistently outperforms state-of-the-art methods on the 300-W, AFLW, and COFW datasets.
  • On the 300-W dataset, GEAN achieved a normalized mean error (NME) of 3.05 on the full test set, outperforming the previous best of 3.12.
  • On the COFW dataset, GEAN achieved 4.24% mean error, surpassing the previous best of 4.40%.
  • The paper demonstrates that increasing the number of branches (manipulated faces) generally improves performance up to a certain point (around five branches), after which the gains diminish relative to the increased computational cost.
  • The framework achieves a speed of 17 FPS on an NVIDIA TITAN X GPU with five branches.

Limitations and Future Work

  • Computational Cost: The generation of manipulated faces and the aggregation process introduce computational overhead. While the framework achieves real-time performance, further optimization might be needed for extremely resource-constrained environments.
  • Trade-off between Speed and Accuracy: The paper notes a trade-off between the speed of manipulation generation (semantic grouping can be slower but more accurate) and the overall framework's speed.
  • Future Work: The authors suggest that the adversarial manipulation step could be decoupled and used to enhance other existing landmark detectors. Further exploration of different manipulation strategies and aggregation techniques could also be beneficial.

Summary updated

BibTeX
@inproceedings{iranmanesh2020robust,
  author = {Seyed Mehdi Iranmanesh and Ali Dabouei and Sobhan Soleymani and Hadi Kazemi and Nasser Nasrabadi},
  title = {{Robust Facial Landmark Detection via Aggregation on Geometrically Manipulated Faces}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure