Am I a Real or Fake Celebrity? Measuring Commercial Face Recognition Web APIs under Deepfake Impersonation Attack

Shahroz Tariq, Sowon Jeon, Simon S. WooarXiv 2021 · arXiv

Abstract

As commercial face recognition APIs become ubiquitous in security and authentication, their robustness against sophisticated spoofing is paramount. This paper presents a comprehensive black-box measurement study of three major commercial APIs—Amazon Rekognition, Microsoft Azure Cognitive Services, and Naver Clova Face Recognition—under Deepfake Impersonation (DI) attacks. Our findings reveal significant vulnerabilities, with non-targeted attack success rates reaching up to 99.9%. We analyze specific API failure modes through novel metrics and propose a practical defense wrapper using a Deep Stacking Ensemble of deepfake detectors, which successfully reduces attack success rates to near zero.

Introduction

Commercial face recognition (FR) APIs are increasingly trusted as gatekeepers for identity verification in diverse applications, ranging from social media tagging to law enforcement. While previous security research has focused heavily on adversarial attacks—adding imperceptible noise to images—this study investigates the more practical and accessible threat of Deepfake Impersonation (DI). In a DI attack, an adversary uses readily available deepfake tools to generate synthetic images of a target victim to bypass celebrity recognition or face similarity checks. This paper evaluates whether current state-of-the-art commercial systems can distinguish between real celebrities and their deepfake counterparts.

Methodology

The researchers conducted a black-box evaluation of Amazon Rekognition, Microsoft Azure, and Naver Clova. The study focused on two primary attack scenarios: Targeted Attacks, where a deepfake of Person A is correctly (but falsely) identified as Person A, and Non-targeted Attacks, where a deepfake is recognized as any valid celebrity, potentially bypassing general liveness checks.

The experimental pipeline illustrating the Deepfake Impersonation Attacker, the targeted commercial APIs, and the evaluation metrics used to assess success rates and fidelity.

To provide a granular analysis of API failures, the authors introduced four novel metrics:

  • Deepfakes with High Fidelity (DHF): Instances where the API assigns a higher confidence score to the deepfake than to the actual real image of the celebrity.
  • Deepfakes with High Confidence (DHC): Cases where the deepfake is accepted with a confidence score exceeding 90%.
  • Deepfakes with High Similarity (DHS): Cases where the API's similarity module scores a fake and real image as >80% similar, even if recognition fails.
  • Successful Impersonation of Celebrities (SIC): A measure of whether at least one deepfake of a specific celebrity successfully fools the API.

Datasets and Deepfake Generation

The study utilized five distinct datasets representing three major categories of deepfake generation: Facial Replacement (face swaps), Facial Reenactment (manipulating expressions), and Facial Synthesis (creating new identities).

Overview of deepfake generation categories: (a) Facial Replacement, (b) Facial Reenactment, and (c) Facial Synthesis, alongside the specific datasets used in the study.

The datasets included public benchmarks like CelebDF and VoxCelebTH, alongside three novel datasets created by the authors: FCelebDF (female-focused face swaps), CelebFOM (using First Order Motion models for reenactment), and CelebBlend (synthetic identities created via FaceApp).

Key Results and API Vulnerabilities

The results demonstrate that commercial APIs are highly susceptible to DI attacks. Targeted attacks reached a maximum success rate of 78.0% (Microsoft API), while non-targeted attacks were nearly universal, reaching 99.9% success against Naver Clova.

Analysis revealed distinct vulnerabilities across providers:

  • Naver Clova was found to be "overly eager," often predicting identities even with low confidence, leading to the highest DHF rates (93.7%).
  • Microsoft Azure was more conservative but proved gullible; when it was fooled, it often assigned extremely high confidence (High DHC).
  • Amazon Rekognition showed significant vulnerability in its similarity module, frequently rating fakes as highly similar to real targets (High DHS).

Furthermore, the study uncovered noticeable racial biases. Western-developed APIs (Amazon, Microsoft) struggled more with Asian celebrities, while the Asian-developed API (Naver) showed a strong bias toward identifying Western deepfakes as Asian actors.

Proposed Defense and Conclusion

To mitigate these risks, the authors proposed a defense wrapper that requires no modification to the underlying commercial APIs. This wrapper utilizes a Deep Stacking Ensemble of off-the-shelf detectors (CLRNet, ABNet, and FTDetect). This ensemble approach proved highly effective, reducing the overall attack success rate from 99.6% to just 1.79%.

In conclusion, while commercial face recognition APIs offer powerful capabilities, they currently lack sufficient robustness against deepfake impersonation. The study highlights the urgent need for integrated deepfake detection and suggests that ensemble-based defense wrappers provide a practical, immediate solution for developers relying on these services. Future work will focus on video-based attacks and generic defense methods like domain adaptation to counter evolving deepfake techniques.

Summary updated

Abstract

Recently, significant advancements have been made in face recognition technologies using Deep Neural Networks. As a result, companies such as Microsoft, Amazon, and Naver offer highly accurate commercial face recognition web services for diverse applications to meet the end-user needs. Naturally, however, such technologies are threatened persistently, as virtually any individual can quickly implement impersonation attacks. In particular, these attacks can be a significant threat for authentication and identification services, which heavily rely on their underlying face recognition technologies' accuracy and robustness. Despite its gravity, the issue regarding deepfake abuse using commercial web APIs and their robustness has not yet been thoroughly investigated. This work provides a measurement study on the robustness of black-box commercial face recognition APIs against Deepfake Impersonation (DI) attacks using celebrity recognition APIs as an example case study. We use five deepfake datasets, two of which are created by us and planned to be released. More specifically, we measure attack performance based on two scenarios (targeted and non-targeted) and further analyze the differing system behaviors using fidelity, confidence, and similarity metrics. Accordingly, we demonstrate how vulnerable face recognition technologies from popular companies are to DI attack, achieving maximum success rates of 78.0% and 99.9% for targeted (i.e., precise match) and non-targeted (i.e., match with any celebrity) attacks, respectively. Moreover, we propose practical defense strategies to mitigate DI attacks, reducing the attack success rates to as low as 0% and 0.02% for targeted and non-targeted attacks, respectively.
BibTeX
@misc{tariq2021am,
  author = {Shahroz Tariq and Sowon Jeon and Simon S. Woo},
  title = {{Am I a Real or Fake Celebrity? Measuring Commercial Face Recognition Web APIs under Deepfake Impersonation Attack}},
  year = {2021},
  eprint = {2103.00847},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.00847},
}
Figure