Accurate and Robust Neural Networks for Security Related Applications Exampled by Face Morphing Attacks

Clemens Seibold, Wojciech Samek, Anna Hilsmann, Peter EisertarXiv 2018 · arXiv

Abstract

This paper investigates the robustness of neural networks in security-critical biometric systems, specifically against face morphing attacks. While Convolutional Neural Networks (CNNs) achieve high accuracy on standard datasets, they often rely on localized shortcuts, making them vulnerable to partial morphs and adversarial perturbations. We propose a multiclass training objective using partial morphs to force the network to learn features across the entire face. Our results show that this approach significantly improves semantic and adversarial robustness, maintaining high detection rates where naive models fail.

Introduction

Face morphing attacks represent a significant threat to biometric systems, such as automated border control. In these attacks, two distinct facial identities are blended into a single synthetic image. If successful, a facial recognition system may match this morphed image to both individuals, allowing an unauthorized person to use a legitimate passport.

While CNNs are highly effective at detecting these forgeries, they often exhibit "lazy" learning behavior. Instead of understanding the global structure of a face, they may focus on specific artifacts (e.g., around the eyes) while ignoring other regions. This makes them fragile: if a forger only morphs the nose or mouth, or applies subtle adversarial perturbations, the network's accuracy collapses. This research aims to force networks to utilize information across the entire facial structure to improve detection robustness.

Figure 1: The face morphing pipeline, illustrating the process from landmark estimation to the final fused image.

Methodology

The study utilized a VGG19 architecture and evaluated four distinct training strategies to combat network laziness:

  1. Naive Training: A baseline using 50% genuine images and 50% complete morphs.
  2. One Region Morphed: Training on images where only one of four specific regions (left eye, right eye, nose, or mouth) is forged.
  3. Complex Morphs: Training on a mixture of genuine images, complete morphs, and partial morphs involving 1 to 4 regions.
  4. Multiclass Training: The network is initially trained to identify which specific region is morphed. This forces the model to build internal representations for every part of the face before being fine-tuned into a binary classifier.

To analyze the decision-making process, the researchers employed Layer-wise Relevance Propagation (LRP). This technique generates heatmaps that highlight the pixels contributing most to a "morphing attack" prediction, revealing whether a network is focusing on the whole face or just specific shortcuts.

Figure 6: Facial regions defined for partial morphing and multiclass training objectives.

Datasets and Preprocessing

The dataset comprised approximately 1,900 individuals sourced from public databases (BU-4DFE, CFD, FEI, FERET, PUT, scFace, and Utrecht) and custom captures. To prevent the network from relying on low-level digital artifacts, various levels of Gaussian noise, salt-and-pepper noise, and blurring were applied to both real and forged images during preprocessing.

Key Results and Insights

Accuracy vs. Robustness

The study found that high accuracy does not equate to robustness. A naively trained model achieved 98% accuracy on standard tests but dropped to a 20% detection rate when faced with a "semantic attack" where only a single facial region was morphed. In contrast, the Multiclass model maintained an 87% detection rate under the same conditions.

Detection Performance

The Multiclass approach achieved the best overall Equal Error Rate (EER) of 2.8%, outperforming the Naive approach (3.1%). LRP heatmaps confirmed that while naive networks focused almost exclusively on the eyes, the multiclass model learned to compare different facial regions against each other, flagging inconsistencies between the eyes, nose, and mouth.

Adversarial Resilience

Against black-box adversarial attacks (Fast Gradient Sign Method), the Multiclass model proved significantly more resilient. With a pixel intensity change of ±6, the Multiclass model correctly identified 91% of attacks, whereas the Naive model's performance degraded rapidly.

Conclusion

By utilizing partial morphs and multiclass objectives, it is possible to "force" neural networks to learn more robust, global features rather than relying on localized shortcuts. While these methods improve resilience against semantic and adversarial attacks, vulnerabilities remain at high perturbation intensities. Future work will explore combining multiclass pre-training with dedicated adversarial defense mechanisms, such as denoising autoencoders, to further harden security models.

Summary updated

Abstract

Artificial neural networks tend to learn only what they need for a task. A manipulation of the training data can counter this phenomenon. In this paper, we study the effect of different alterations of the training data, which limit the amount and position of information that is available for the decision making. We analyze the accuracy and robustness against semantic and black box attacks on the networks that were trained on different training data modifications for the particular example of morphing attacks. A morphing attack is an attack on a biometric facial recognition system where the system is fooled to match two different individuals with the same synthetic face image. Such a synthetic image can be created by aligning and blending images of the two individuals that should be matched with this image.
BibTeX
@misc{seibold2018accurate,
  author = {Clemens Seibold and Wojciech Samek and Anna Hilsmann and Peter Eisert},
  title = {{Accurate and Robust Neural Networks for Security Related Applications Exampled by Face Morphing Attacks}},
  year = {2018},
  eprint = {1806.04265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.04265},
}
Figure