Abstract Advancements in machine learning and deep learning techniques have led to the development of sophisticated and accurate face recognition systems. However, researchers are increasingly exploring the vulnerabilities of these systems to digital attacks. The creation of digitally altered images has become trivial with modern editing tools and mobile applications. This research proposes a "partial face tampering" attack, where specific facial regions are replaced or morphed to generate tampered samples. Experiments using state-of-the-art face recognition systems like VGG-Face and OpenFace on the CMU-MultiPIE dataset demonstrate significant vulnerabilities. To counter this, we propose a Partial Face Tampering Detection (PFTD) network that combines raw RGB and high-frequency information to identify inconsistencies, outperforming existing baseline models.
Introduction
Face recognition technology is now ubiquitous, securing everything from smartphones to border crossings. While full-face morphing and adversarial noise are well-documented threats, "partial" tampering represents a more subtle and accessible vulnerability. Using common tools like Photoshop or Snapchat, an attacker can surgically swap or blend specific features—such as the eyes, nose, or mouth—from a different individual. This paper investigates whether deep face recognition systems are robust to these minuscule, localized changes that often remain visually convincing to human observers.
The Attack: Partial Face Tampering
The authors introduce two primary methods for generating tampered samples:
- Replacement of Facial Regions (RFR): This method uses the Viola-Jones detector to isolate specific features (eyes, nose, or mouth). These regions are then replaced with corresponding features from a source image, with Gaussian filtering applied to smooth the boundaries.
- Morphing of Facial Regions (MFR): Instead of direct replacement, features from two subjects are blended using varying proportions (e.g., 40% or 50% blending) to create a hybrid identity.
The core insight is that deep networks often rely heavily on specific facial "anchor" points. By tampering with just one key region, an attacker can throw the AI's internal feature weights into disarray while maintaining the global structure of the face, effectively fooling the system.
The Defense: Partial Face Tampering Detection (PFTD)
To detect these subtle alterations, the researchers developed a two-stream Convolutional Neural Network (CNN) based on the VGG-Face architecture:
- RGB Stream: Processes the raw image to identify inconsistencies in contrast and boundary "seams" introduced during the tampering process.
- High-Pass Filter Stream: Focuses on the residual domain. Smoothing operations used to hide tampering often leave artifacts in high-frequency textures that are invisible in standard RGB but detectable here.
The features from both streams are fused and passed through fully connected layers for final classification.
Experimental Results
The vulnerability of existing systems is stark. When subjected to an RFR attack on the eye region, the Genuine Accept Rate (GAR) for OpenFace plummeted from 85.91% to 34.89%. VGG-Face saw a similar drop from 99.97% to 53.59%.
Conversely, the proposed PFTD network demonstrated high efficacy, achieving a 91.44% detection accuracy. This significantly outperformed standard baselines like VGG16 (54.06%) and fine-tuned VGG-Face models (81.78%). Furthermore, the PFTD model showed strong cross-dataset robustness, maintaining a 93.69% accuracy on low-quality GAN-generated videos from the DeepFake database, despite not being specifically trained on them.
Conclusion
Partial face tampering poses a significant threat to modern face recognition systems by exploiting their reliance on specific facial regions. While these "surgical" attacks are difficult for humans to perceive, the proposed PFTD network effectively leverages high-frequency information to detect them. Future work will focus on localizing the tampered regions and improving robustness against "in-the-wild" conditions such as extreme poses and motion blur.
Summary updated