A Survey on Physical Adversarial Attacks against Face Recognition Systems

Mingsi Wang, Jiachen Zhou, Tianlin Li, Guozhu Meng, Kai ChenarXiv 2024 · DOI · arXiv

Abstract

Face Recognition (FR) systems are increasingly deployed in critical security applications, from border control to mobile authentication. While digital adversarial attacks are well-studied, physical adversarial attacks—which manipulate the environment or the subject's appearance before image capture—present a more realistic and severe threat. This paper provides a systematic survey of physical adversarial attacks, categorizing them by physical medium and optimization strategy. We explore the "virtual-to-real-to-virtual" (V2R2V) pipeline, the trade-offs between robustness and imperceptibility, and the current state of defenses, providing a roadmap for future research in biometric security.

Introduction: The Physical Threat Model

While digital adversarial attacks (directly modifying image pixels) are well-documented, they are often impractical in real-world scenarios because an attacker rarely has access to the digital pipeline of a security camera. Physical adversarial attacks—where an attacker wears a specific accessory or uses light to deceive a camera—pose a much more significant threat to systems used in banking, border control, and smartphone unlocking.

The pipeline of face recognition systems and the entry points for physical vs. digital adversarial attacks.

The authors address a major gap in the literature: the lack of a structured overview that clarifies the unique challenges of the "virtual-to-real-to-virtual" (V2R2V) process. In this process, an adversarial pattern must be designed digitally, manufactured physically (printed or projected), and then recaptured by a camera, surviving environmental noise, lighting changes, and hardware limitations along the way.

Taxonomy of Physical Attacks

The paper categorizes physical attacks into three main domains based on the medium used, as shown in the taxonomy below:

Taxonomy of physical adversarial attacks categorized by medium: disguise-based, infrared-based, and illumination-based.
  1. Disguise-based: Using physical objects like 3D-printed masks, adversarial glasses (e.g., AdvGlass), stickers (AdvSticker), or even carefully applied makeup (AdvMakeup).
  2. Infrared-based: Exploiting the camera’s sensitivity to light invisible to the human eye using infrared LEDs or lasers to disrupt facial feature extraction.
  3. Illumination-based: Projecting adversarial patterns onto a face or using "adversarial relighting" to change how a model perceives facial geometry.

Optimization Strategies

The authors distinguish between two primary optimization strategies used to generate these physical perturbations:

  • Pixel Space Strategy: Using gradient descent to optimize the raw values of a patch. This often requires techniques like Expectation over Transformation (EoT) to ensure the patch remains effective across different angles and Non-Printable Score (NPS) to ensure the colors can be accurately reproduced by a printer.
  • Latent Space Strategy: Leveraging Generative Adversarial Networks (GANs) to create perturbations that look like natural textures (e.g., skin or makeup), making them much harder for humans to detect.

Datasets and Evaluation

The survey reviews several standard benchmarks used to evaluate these attacks. 2D Datasets like LFW, CelebA, and VGGFace2 are the most common for training. 3D Datasets such as Bosphorus and SIAT-3DFE are used for advanced attacks accounting for facial depth. Evaluation settings are split between controlled laboratory environments and uncontrolled real-world tests against commercial APIs like Face++ or smartphone face-unlock features.

Key Insights and Results

  • The Robustness-Imperceptibility Trade-off: The most effective attacks (like high-contrast stickers) are often obvious to humans. Stealthy attacks (like infrared) are easily countered by modern hardware like infrared cutoff filters.
  • Stickers as the "Sweet Spot": Unlike glasses or masks with fixed positions, adversarial stickers allow for simultaneous optimization of both the perturbation and the location on the face, providing high flexibility.
  • Dodging vs. Impersonation: "Dodging" (making the system fail to recognize the subject) is significantly easier to achieve than "Impersonation" (making the system identify the subject as a specific target).
  • Digital-to-Physical Loss: There is a consistent drop in effectiveness when moving from digital simulations to physical implementation, often due to "chromatic aberration" (color shifts) from printers and cameras.

Future Directions

The authors identify several critical areas for future research:

  • Standardization: There is currently no unified benchmark for physical attacks, making it difficult to compare results across different hardware and lighting conditions.
  • Better Metrics: Current metrics like L2 norm don't capture human-like perceptibility. The authors suggest using Visual Large Language Models (VLLMs) to automate this assessment.
  • Diffusion Models: While GANs are the current standard, Diffusion Models (like SDedit) represent the next frontier for generating realistic physical perturbations.
  • Multimodal Defense: Future FR systems should integrate depth information, infrared, and gait analysis to identify physical artifacts that a 2D camera might miss.

Summary updated

Abstract

As Face Recognition (FR) technology becomes increasingly prevalent in finance, the military, public safety, and everyday life, security concerns have grown substantially. Physical adversarial attacks targeting FR systems in real-world settings have attracted considerable research interest due to their practicality and the severe threats they pose. However, a systematic overview focused on physical adversarial attacks against FR systems is still lacking, hindering an in-depth exploration of the challenges and future directions in this field. In this paper, we bridge this gap by comprehensively collecting and analyzing physical adversarial attack methods targeting FR systems. Specifically, we first investigate the key challenges of physical attacks on FR systems. We then categorize existing physical attacks into three categories based on the physical medium used and summarize how the research in each category has evolved to address these challenges. Furthermore, we review current defense strategies and discuss potential future research directions. Our goal is to provide a fresh, comprehensive, and deep understanding of physical adversarial attacks against FR systems, thereby inspiring relevant research in this area.
BibTeX
@misc{wang2024survey,
  author = {Mingsi Wang and Jiachen Zhou and Tianlin Li and Guozhu Meng and Kai Chen},
  title = {{A Survey on Physical Adversarial Attacks against Face Recognition Systems}},
  year = {2024},
  eprint = {2410.16317},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.16317},
}
Figure