Abstract
Face Recognition (FR) systems are increasingly deployed in critical security applications, from border control to mobile authentication. While digital adversarial attacks are well-studied, physical adversarial attacks—which manipulate the environment or the subject's appearance before image capture—present a more realistic and severe threat. This paper provides a systematic survey of physical adversarial attacks, categorizing them by physical medium and optimization strategy. We explore the "virtual-to-real-to-virtual" (V2R2V) pipeline, the trade-offs between robustness and imperceptibility, and the current state of defenses, providing a roadmap for future research in biometric security.
Introduction: The Physical Threat Model
While digital adversarial attacks (directly modifying image pixels) are well-documented, they are often impractical in real-world scenarios because an attacker rarely has access to the digital pipeline of a security camera. Physical adversarial attacks—where an attacker wears a specific accessory or uses light to deceive a camera—pose a much more significant threat to systems used in banking, border control, and smartphone unlocking.
The authors address a major gap in the literature: the lack of a structured overview that clarifies the unique challenges of the "virtual-to-real-to-virtual" (V2R2V) process. In this process, an adversarial pattern must be designed digitally, manufactured physically (printed or projected), and then recaptured by a camera, surviving environmental noise, lighting changes, and hardware limitations along the way.
Taxonomy of Physical Attacks
The paper categorizes physical attacks into three main domains based on the medium used, as shown in the taxonomy below:
- Disguise-based: Using physical objects like 3D-printed masks, adversarial glasses (e.g., AdvGlass), stickers (AdvSticker), or even carefully applied makeup (AdvMakeup).
- Infrared-based: Exploiting the camera’s sensitivity to light invisible to the human eye using infrared LEDs or lasers to disrupt facial feature extraction.
- Illumination-based: Projecting adversarial patterns onto a face or using "adversarial relighting" to change how a model perceives facial geometry.
Optimization Strategies
The authors distinguish between two primary optimization strategies used to generate these physical perturbations:
- Pixel Space Strategy: Using gradient descent to optimize the raw values of a patch. This often requires techniques like Expectation over Transformation (EoT) to ensure the patch remains effective across different angles and Non-Printable Score (NPS) to ensure the colors can be accurately reproduced by a printer.
- Latent Space Strategy: Leveraging Generative Adversarial Networks (GANs) to create perturbations that look like natural textures (e.g., skin or makeup), making them much harder for humans to detect.
Datasets and Evaluation
The survey reviews several standard benchmarks used to evaluate these attacks. 2D Datasets like LFW, CelebA, and VGGFace2 are the most common for training. 3D Datasets such as Bosphorus and SIAT-3DFE are used for advanced attacks accounting for facial depth. Evaluation settings are split between controlled laboratory environments and uncontrolled real-world tests against commercial APIs like Face++ or smartphone face-unlock features.
Key Insights and Results
- The Robustness-Imperceptibility Trade-off: The most effective attacks (like high-contrast stickers) are often obvious to humans. Stealthy attacks (like infrared) are easily countered by modern hardware like infrared cutoff filters.
- Stickers as the "Sweet Spot": Unlike glasses or masks with fixed positions, adversarial stickers allow for simultaneous optimization of both the perturbation and the location on the face, providing high flexibility.
- Dodging vs. Impersonation: "Dodging" (making the system fail to recognize the subject) is significantly easier to achieve than "Impersonation" (making the system identify the subject as a specific target).
- Digital-to-Physical Loss: There is a consistent drop in effectiveness when moving from digital simulations to physical implementation, often due to "chromatic aberration" (color shifts) from printers and cameras.
Future Directions
The authors identify several critical areas for future research:
- Standardization: There is currently no unified benchmark for physical attacks, making it difficult to compare results across different hardware and lighting conditions.
- Better Metrics: Current metrics like L2 norm don't capture human-like perceptibility. The authors suggest using Visual Large Language Models (VLLMs) to automate this assessment.
- Diffusion Models: While GANs are the current standard, Diffusion Models (like SDedit) represent the next frontier for generating realistic physical perturbations.
- Multimodal Defense: Future FR systems should integrate depth information, infrared, and gait analysis to identify physical artifacts that a 2D camera might miss.
Summary updated