Evading Face Recognition via Partial Tampering of Faces

Puspita Majumdar, Akshay Agarwal, Richa Singh, Mayank VatsaWorkshops @ CVPR 2019 · OpenReview

Abstract Advancements in machine learning and deep learning techniques have led to the development of sophisticated and accurate face recognition systems. However, researchers are increasingly exploring the vulnerabilities of these systems to digital attacks. The creation of digitally altered images has become trivial with modern editing tools and mobile applications. This research proposes a "partial face tampering" attack, where specific facial regions are replaced or morphed to generate tampered samples. Experiments using state-of-the-art face recognition systems like VGG-Face and OpenFace on the CMU-MultiPIE dataset demonstrate significant vulnerabilities. To counter this, we propose a Partial Face Tampering Detection (PFTD) network that combines raw RGB and high-frequency information to identify inconsistencies, outperforming existing baseline models.

Introduction

Face recognition technology is now ubiquitous, securing everything from smartphones to border crossings. While full-face morphing and adversarial noise are well-documented threats, "partial" tampering represents a more subtle and accessible vulnerability. Using common tools like Photoshop or Snapchat, an attacker can surgically swap or blend specific features—such as the eyes, nose, or mouth—from a different individual. This paper investigates whether deep face recognition systems are robust to these minuscule, localized changes that often remain visually convincing to human observers.

The Attack: Partial Face Tampering

The authors introduce two primary methods for generating tampered samples:

  1. Replacement of Facial Regions (RFR): This method uses the Viola-Jones detector to isolate specific features (eyes, nose, or mouth). These regions are then replaced with corresponding features from a source image, with Gaussian filtering applied to smooth the boundaries.
  2. Morphing of Facial Regions (MFR): Instead of direct replacement, features from two subjects are blended using varying proportions (e.g., 40% or 50% blending) to create a hybrid identity.
Figure 1: Examples of original and partially tampered face images from the Multi-PIE dataset.

The core insight is that deep networks often rely heavily on specific facial "anchor" points. By tampering with just one key region, an attacker can throw the AI's internal feature weights into disarray while maintaining the global structure of the face, effectively fooling the system.

The Defense: Partial Face Tampering Detection (PFTD)

To detect these subtle alterations, the researchers developed a two-stream Convolutional Neural Network (CNN) based on the VGG-Face architecture:

  • RGB Stream: Processes the raw image to identify inconsistencies in contrast and boundary "seams" introduced during the tampering process.
  • High-Pass Filter Stream: Focuses on the residual domain. Smoothing operations used to hide tampering often leave artifacts in high-frequency textures that are invisible in standard RGB but detectable here.

The features from both streams are fused and passed through fully connected layers for final classification.

Experimental Results

The vulnerability of existing systems is stark. When subjected to an RFR attack on the eye region, the Genuine Accept Rate (GAR) for OpenFace plummeted from 85.91% to 34.89%. VGG-Face saw a similar drop from 99.97% to 53.59%.

Conversely, the proposed PFTD network demonstrated high efficacy, achieving a 91.44% detection accuracy. This significantly outperformed standard baselines like VGG16 (54.06%) and fine-tuned VGG-Face models (81.78%). Furthermore, the PFTD model showed strong cross-dataset robustness, maintaining a 93.69% accuracy on low-quality GAN-generated videos from the DeepFake database, despite not being specifically trained on them.

Conclusion

Partial face tampering poses a significant threat to modern face recognition systems by exploiting their reliance on specific facial regions. While these "surgical" attacks are difficult for humans to perceive, the proposed PFTD network effectively leverages high-frequency information to detect them. Future work will focus on localizing the tampered regions and improving robustness against "in-the-wild" conditions such as extreme poses and motion blur.

Summary updated

Abstract

Advancements in machine learning and deep learning techniques have led to the development of sophisticated and accurate face recognition systems. However, for the past few years, researchers are exploring the vulnerabilities of these systems towards digital attacks. Creation of digitally altered images has become an easy task with the availability of various image editing tools and mobile application such as Snapchat. Morphing based digital attacks are used to elude and gain the identity of legitimate users by fooling the deep networks. In this research, partial face tampering attack is proposed, where facial regions are replaced or morphed to generate tampered samples. Face verification experiments performed using two state-of-the-art face recognition systems, VGG-Face and OpenFace on the CMU- MultiPIE dataset indicates the vulnerability of these systems towards the attack. Further, a Partial Face Tampering Detection (PFTD) network is proposed for the detection of the proposed attack. The network captures the inconsistencies among the original and tampered images by combining the raw and high-frequency information of the input images for the detection of tampered images. The proposed network surpasses the performance of the existing baseline deep neural networks for tampered image detection.
BibTeX
@inproceedings{majumdar2019evading,
  author = {Puspita Majumdar and Akshay Agarwal and Richa Singh and Mayank Vatsa},
  title = {{Evading Face Recognition via Partial Tampering of Faces}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=SOUVTZGzer2T},
}
Figure