Geometrically Adaptive Dictionary Attack on Face Recognition

Junyoung Byun, Hyojun Go, Changick Kim

Abstract

CNN-based face recognition models have brought remarkable performance improvement, but they are vulnerable to adversarial perturbations. Recent studies have shown that adversaries can fool the models even if they can only access the models’ hard-label output. However, since many queries are needed to find imperceptible adversarial noise, reducing the number of queries is crucial for these attacks. In this paper, we point out two limitations of existing decision-based black-box attacks. We observe that they waste queries for background noise optimization, and they do not take advantage of adversarial perturbations generated for other images. We exploit 3D face alignment to overcome these limitations and propose a general strategy for query-efficient black-box attacks on face recognition named Geometrically Adaptive Dictionary Attack (GADA). Our core idea is to create an adversarial perturbation in the UV texture map and project it onto the face in the image. It greatly improves query efficiency by limiting the perturbation search space to the facial area and effectively recycling previous perturbations. We apply the GADA strategy to two existing attack methods and show overwhelming performance improvement in the experiments on the LFW and CPLFW datasets. Furthermore, we also present a novel attack strategy that can circumvent query similarity-based stateful detection that identifies the process of query-based black-box attacks.

Introduction

Adversarial attacks—subtle tweaks to an image that cause AI to misclassify it—are a major security concern for face recognition systems used in payments and law enforcement. In a "black-box" scenario, the attacker doesn't know the model's internal architecture or weights; they can only observe the final output (e.g., "Access Granted" or "Denied").

Current black-box attacks face two primary challenges:

  1. Query Inefficiency: Existing methods often attempt to optimize every pixel in an image, including the background. Since face recognition models primarily focus on facial features, this results in significant query waste.
  2. Lack of Transferability/Memory: Most attacks start from scratch for every new target image, failing to leverage successful perturbation patterns discovered in previous attacks.

Because these attacks require thousands of queries to find an imperceptible perturbation, they are slow, computationally expensive, and easily detected by system administrators.

Methodology: Geometrically Adaptive Dictionary Attack (GADA)

To address these limitations, we introduce GADA, which shifts the attack from 2D pixel space to a 3D geometric space.

3D Face Alignment and UV Mapping

GADA utilizes a pre-trained 3D face alignment model (e.g., 3DDFA_V2) to estimate the 3D shape of the face within a 2D image. Instead of adding noise directly to the $(x, y)$ coordinates of the image, GADA operates on a UV texture map—a flattened, 2D representation of the 3D face surface. This noise is then "wrapped" back onto the 3D face and projected into the final image.

Geometrically adaptive perturbation restricted to the facial region.

By restricting the perturbation search space to the facial area (via the 3D mask), GADA naturally filters out "dead weight" pixels in the background. This allows the optimization algorithm to converge much faster.

Dictionary Attacks

The UV map provides a consistent semantic space; for instance, pixels corresponding to the "nose" are always located in the same region of the UV map, regardless of head pose. We exploit this by creating a "dictionary" of successful perturbations. When attacking a new face, GADA retrieves a similar perturbation from the dictionary to serve as a high-quality "warm start," significantly reducing the number of queries needed to achieve success.

Evading Stateful Detection

To bypass "stateful detectors" that flag users sending many similar-looking images, GADA employs a strategy of injecting random noise into the background while maintaining a precise facial perturbation. This breaks the perceptual similarity that detectors look for without compromising the attack's effectiveness.

Experimental Evaluation

We evaluated GADA using two standard benchmarks: LFW (Labeled Faces in the Wild) and CPLFW (Cross-Pose LFW). The latter specifically tests the robustness of the 3D alignment under significant variations in head pose.

Key Results

Applying the GADA strategy to existing frameworks like Evolutionary Attack (EA) and Sign-Flip Attack (SFA) yielded substantial improvements:

  • Efficiency: On the LFW dataset, the GADA-enhanced attack (EAGD) achieved the same level of perturbation invisibility as the baseline while requiring 2,100 fewer queries on average.
  • Performance at Scale: With a limited budget of 1,000 queries, GADA nearly halved the perturbation norm compared to standard EA, making the resulting adversarial images much harder for humans to detect.
An adversarial image generated using the GADA strategy, showing subtle facial noise.

Conclusion and Future Work

GADA demonstrates that semantic correspondence and search space reduction are critical for efficient black-box attacks. By leveraging 3D geometry, we can recycle adversarial patterns across different individuals and poses.

However, the method's success depends on the accuracy of the underlying 3D face alignment. Future work could explore more robust 3D mesh estimators to handle extreme occlusions or extend this dictionary-based approach to general object recognition by developing 3D templates for other categories.

Summary updated

Abstract

CNN-based face recognition models have brought remarkable performance improvement, but they are vulnerable to adversarial perturbations. Recent studies have shown that adversaries can fool the models even if they can only access the models’ hard-label output. However, since many queries are needed to find imperceptible adversarial noise, reducing the number of queries is crucial for these attacks. In this paper, we point out two limitations of existing decision-based black-box attacks. We observe that they waste queries for background noise optimization, and they do not take advantage of adversarial perturbations generated for other images. We exploit 3D face alignment to overcome these limitations and propose a general strategy for query-efficient black-box attacks on face recognition named Geometrically Adaptive Dictionary Attack (GADA). Our core idea is to create an adversarial perturbation in the UV texture map and project it onto the face in the image. It greatly improves query efficiency by limiting the perturbation search space to the facial area and effectively recycling previous perturbations. We apply the GADA strategy to two existing attack methods and show overwhelming performance improvement in the experiments on the LFW and CPLFW datasets. Furthermore, we also present a novel attack strategy that can circumvent query similarity-based stateful detection that identifies the process of query-based black-box attacks.
BibTeX
@inproceedings{byun2022geometrically,
  author = {Junyoung Byun and Hyojun Go and Changick Kim},
  title = {{Geometrically Adaptive Dictionary Attack on Face Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=YxDpfnVCFs},
}
Figure