Geometrically Adaptive Dictionary Attack on Face Recognition

Junyoung Byun, Hyojun Go, Changick Kim

Abstract

CNN-based face recognition models have brought remarkable performance improvement, but they are vulnerable to adversarial perturbations. Recent studies have shown that adversaries can fool the models even if they can only access the models' hard-label output. However, since many queries are needed to find imperceptible adversarial noise, reducing the number of queries is crucial for these attacks. In this paper, we point out two limitations of existing decision-based black-box attacks. We observe that they waste queries for background noise optimization, and they do not take advantage of adversarial perturbations generated for other images. We exploit 3D face alignment to overcome these limitations and propose a general strategy for query-efficient black-box attacks on face recognition named Geometrically Adaptive Dictionary Attack (GADA). Our core idea is to create an adversarial perturbation in the UV texture map and project it onto the face in the image. It greatly improves query efficiency by limiting the perturbation search space to the facial area and effectively recycling previous perturbations. We apply the GADA strategy to two existing attack methods and show overwhelming performance improvement in the experiments on the LFW and CPLFW datasets. Furthermore, we also present a novel attack strategy that can circumvent query similarity-based stateful detection that identifies the process of query-based black-box attacks.

Introduction

Adversarial attacks on face recognition systems pose a significant security threat. In black-box scenarios, where attackers only have access to hard-label outputs (e.g., "Match" or "No Match"), finding imperceptible noise typically requires thousands of queries. This high query count makes attacks slow, computationally expensive, and easy to detect by stateful security systems. Existing methods often suffer from two main inefficiencies: they optimize noise in irrelevant background areas and fail to leverage knowledge from previous successful attacks on different subjects.

Methodology: Geometrically Adaptive Dictionary Attack (GADA)

The proposed GADA framework addresses these inefficiencies by shifting the attack from the 2D pixel domain to a 3D geometric space.

3D UV Mapping

Instead of applying perturbations to the entire 2D image, GADA utilizes a 3D Morphable Model (3DMM) to align the face and "unwrap" it into a standardized 2D UV texture map. Adversarial noise is then optimized directly on this UV map. By projecting this noise back onto the 3D face model in the image, the attack focuses exclusively on the facial region, which is the primary area of interest for recognition models. This geometric prior significantly reduces the search space for adversarial noise.

Comparison between standard image-domain attacks and the proposed UV-domain attack (GADA).

Dictionary Attack and Recycling

A key innovation of GADA is the "Dictionary Attack." Because the UV map provides a canonical representation of a face, adversarial perturbations generated for one individual can often serve as effective starting points for attacking another. GADA maintains a dictionary of successful perturbations, retrieving and applying them based on facial similarity to jump-start new attacks, drastically reducing the number of queries needed to reach convergence.

Adversarial perturbation generated in the UV texture domain.

Evasion of Stateful Detection

To bypass security monitors that detect sequences of highly similar queries, GADA employs a background-noise-injection strategy. By adding random noise to the non-facial background while maintaining the adversarial signal on the face, the attack remains effective against the face recognition model (which ignores the background) while appearing as a series of distinct images to a whole-image similarity detector.

Experimental Results

The effectiveness of GADA was evaluated on the LFW and CPLFW datasets against baseline attacks such as Evolutionary Attack (EA) and HopSkipJumpAttack (HSJA).

  • Query Efficiency: On the LFW dataset, GADA achieved the target perturbation norm using approximately 2,100 fewer queries than the baseline EA attack.
  • Attack Performance: Under a fixed budget of 1,000 queries, GADA produced perturbations with nearly half the norm of competing methods, making the noise significantly less perceptible.
  • Stealth: While standard attacks were caught 100% of the time by similarity-based detectors, GADA's background noise strategy successfully bypassed these systems.

Conclusion

GADA demonstrates that incorporating geometric priors through 3D face alignment can significantly enhance the efficiency and stealth of black-box adversarial attacks. By focusing on the UV texture domain, attackers can recycle successful noise patterns and avoid wasting queries on irrelevant background pixels. Future work may explore extending this region-of-interest approach to other object categories through semantic alignment.

Summary updated

Abstract

CNN-based face recognition models have brought remarkable performance improvement, but they are vulnerable to adversarial perturbations. Recent studies have shown that adversaries can fool the models even if they can only access the models' hard-label output. However, since many queries are needed to find imperceptible adversarial noise, reducing the number of queries is crucial for these attacks. In this paper, we point out two limitations of existing decision-based black-box attacks. We observe that they waste queries for background noise optimization, and they do not take advantage of adversarial perturbations generated for other images. We exploit 3D face alignment to overcome these limitations and propose a general strategy for query-efficient black-box attacks on face recognition named Geometrically Adaptive Dictionary Attack (GADA). Our core idea is to create an adversarial perturbation in the UV texture map and project it onto the face in the image. It greatly improves query efficiency by limiting the perturbation search space to the facial area and effectively recycling previous perturbations. We apply the GADA strategy to two existing attack methods and show overwhelming performance improvement in the experiments on the LFW and CPLFW datasets. Furthermore, we also present a novel attack strategy that can circumvent query similarity-based stateful detection that identifies the process of query-based black-box attacks.
BibTeX
@inproceedings{byun2022geometrically,
  author = {Junyoung Byun and Hyojun Go and Changick Kim},
  title = {{Geometrically Adaptive Dictionary Attack on Face Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=YxDpfnVCFs},
}
Figure