OPOM: Customized Invisible Cloak Towards Face Privacy Protection

Yaoyao Zhong, Weihong Deng

TL;DR

Researchers have developed OPOM (One Person One Mask), a method to create a single, person-specific "invisible cloak" that protects all of a user's photos and videos from unauthorized face recognition. By generating one adversarial mask per identity rather than per image, the system offers a high-efficiency, high-protection middle ground between slow image-specific filters and weak general-purpose masks.

What Problem Does This Paper Address?

While face recognition (FR) offers convenience, it also enables surreptitious surveillance and unauthorized data harvesting from social media. Current privacy-preserving "masks"—imperceptible digital noise added to images to fool AI—suffer from a usability gap:

  1. Image-specific masks (like Fawkes or LowKey) are highly effective but computationally expensive, requiring a new mask to be generated for every single frame of a video or every new photo uploaded.
  2. Universal masks are fast but aim to fool FR systems for everyone at once, which often results in lower protection rates for specific individuals.

The authors address this by proposing a person-specific (class-wise) universal mask. A user generates their custom "cloak" once using a few reference images and can then apply that same cloak to every future photo or video they share.

Methods

The core of the paper is the OPOM (One Person One Mask) framework. The goal is to find a perturbation ($\Delta X$) that, when added to any image of person $k$, pushes that image's feature vector away from the "feature subspace" that represents their identity in an FR model.

The researchers explored three ways to model this "identity subspace" using only a small number of training images:

  • Class Centers: Representing the identity as a single average point.
  • Affine Hulls: Treating any linear combination of the user's features as a valid representation of their identity.
  • Convex Hulls: Modeling the identity as the smallest convex set containing all training features.

OPOM-ConvexHull emerged as the most effective method. To ensure these masks work against "black-box" systems (FR models the user doesn't have access to), the authors integrated transferability techniques like Momentum Boosting and DFANet (which uses dropout to simulate a variety of different network architectures during mask generation).

Datasets

The authors evaluated their method on three primary benchmarks:

  • Privacy-Commons: 500 identities from the MegaFace dataset, using 10 images for training and 5 for testing.
  • Privacy-Celebrities: 500 identities from a million-celebrity list, using 10 images for training and 10 for testing.
  • Sherlock: A video dataset used for a practical application study, featuring characters from the TV series Sherlock (Benedict Cumberbatch and Martin Freeman).

Key Insights and Ideas

  • The "Region" vs. "Point" Advantage: Modeling an identity as a Convex Hull (a region) rather than a single point (Class Center) significantly improves protection. It provides more diverse gradient information during the optimization process, making the resulting mask more robust to changes in pose or lighting.
  • Efficiency is the Killer App: Unlike image-specific methods that take seconds per image, OPOM has a 0-second generation time for new images once the initial mask is created. This makes it viable for real-time video protection.
  • The Transferability Gap: The paper highlights that masks generated on one model (e.g., ArcFace) can still protect against entirely different architectures (e.g., MobileNet or SENet) and even commercial "black-box" APIs.

Key Results

  • High Protection Success: On the Privacy-Commons dataset, OPOM-ConvexHull achieved a Top-1 protection success rate of ~86% against black-box models when the source model was ArcFace.
  • Commercial API Defeat: The masks significantly reduced the similarity scores of the same person on major commercial platforms, including Amazon Rekognition, Microsoft Azure, Baidu Cloud, and Face++.
  • Video Practicality: In the Sherlock case study, applying a single person-specific mask to video frames successfully prevented the characters from being recognized, even as their poses and expressions changed throughout the scene.

Limitations and Future Work

  • Extreme Variance: The authors acknowledge that if a test image has a drastically different pose, illumination, or occlusion compared to the training samples, the mask's protection can "break down."
  • Mask Leakage: If a malicious FR system manages to collect enough "masked" images and labels them, it might eventually learn to recognize the masked version of the user. The authors began exploring "diverse masks" (generating multiple different masks for one person) to counter this, but it remains a preliminary area of research.
  • Digital-Only: Currently, this is a digital "cloak." Future work would be needed to translate these adversarial patterns into physical items (like glasses or clothing) that work in the real world.

Summary updated

Abstract

While convenient in daily life, face recognition technologies also raise privacy concerns for regular users on the social media since they could be used to analyze face images and videos, efficiently and surreptitiously without any security restrictions. In this paper, we investigate the face privacy protection from a technology standpoint based on a new type of customized cloak, which can be applied to all the images of a regular user, to prevent malicious face recognition systems from uncovering their identity. Specifically, we propose a new method, named one person one mask (OPOM), to generate person-specific (class-wise) universal masks by optimizing each training sample in the direction away from the feature subspace of the source identity. To make full use of the limited training images, we investigate several modeling methods, including affine hulls, class centers and convex hulls, to obtain a better description of the feature subspace of source identities. The effectiveness of the proposed method is evaluated on both common and celebrity datasets against black-box face recognition models with different loss functions and network architectures. In addition, we discuss the advantages and potential problems of the proposed method. In particular, we conduct an application study on the privacy protection of a video dataset, Sherlock, to demonstrate the potential practical usage of the proposed method.
BibTeX
@article{zhong2023opom,
  author = {Yaoyao Zhong and Weihong Deng},
  title = {{OPOM: Customized Invisible Cloak Towards Face Privacy Protection}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=euz_xV3K1hu},
}

Loading related content...

Figure