Learning Privacy Preserving Encodings through Adversarial Training

Francesco Pittaluga, Sanjeev J. Koppal, Ayan Chakrabarti

Introduction

As visual data sharing becomes ubiquitous, the risk of leaking sensitive information—such as identities, locations, or private activities—increases. While traditional methods like blurring or pixelation are commonly used, modern deep neural networks (DNNs) are remarkably proficient at identifying "redundant cues" that persist even in degraded images. This paper addresses the challenge of creating image encodings that scrub specific private attributes while maintaining the utility of the data for desirable tasks.

A critical distinction made by the authors is between transient and persistent privacy. In standard adversarial setups, an encoder might successfully "fool" a classifier during joint training. However, once the encoder is fixed, a high-capacity adversary can often retrain a new classifier on the encoded data to recover the private information. The proposed framework aims for persistent privacy, ensuring that private information remains unrecoverable even after exhaustive retraining by an adversary.

Methodology

The framework employs an adversarial optimization loop where an Encoder ($E$) is pitted against a Private Attribute Classifier ($\Phi_u$). To ensure the encoding remains useful, the encoder also works with a Desirable Attribute Classifier ($\Phi_v$). Two primary innovations drive the success of this approach:

  1. Label Flip Loss: Standard GAN losses often focus on minimizing classifier confidence, which can lead to unstable training. Instead, the authors use a "label flip" loss that trains the encoder to drive the adversary's prediction toward the opposite of the true label. This forces the encoder to actively remove features that allow for class separation.
  2. Per-location Normalization: To prevent the common failure mode where an encoder "collapses" to a constant output to achieve privacy, the authors implement a normalization layer. This layer enforces zero mean and unit variance at every spatial location across a batch, ensuring the encoder produces diverse, information-rich outputs.

Experimental Results

The researchers evaluated the framework on the Places-365 dataset, focusing on binary classification tasks (e.g., detecting an "Army Base" vs. an "Airport Terminal").

Persistent Privacy and Utility

The most rigorous test of the system is the "cat-and-mouse" verification protocol. After the adversarial training phase, the encoder is fixed, and a new classifier is trained from scratch to its saturation point.

Figure 1: Training curves showing classifier accuracy over time for private and desirable tasks. The proposed method (green) significantly inhibits the private task compared to standard GAN updates (yellow) and blurring (blue), while maintaining high accuracy on the desirable task.

As shown in Figure 1, the proposed "Label Flip" method maintains a significantly lower accuracy for the private task (e.g., 64.2% for Army Base detection) compared to standard GAN-based adversarial training (83.9%). Crucially, this privacy does not come at the cost of utility; the encoder trained to hide "Army Base" while promoting "Airport Terminal" achieved 95.6% accuracy on the desirable task, outperforming the naive blur baseline.

Qualitative Analysis

The resulting encodings do not resemble natural images. Instead, they appear as high-dimensional feature maps that are difficult for humans to interpret but remain highly effective for trained machine learning models.

Figure 3: Visual comparison of original images, blurred baselines, and learned encodings. The learned encodings effectively scrub private attributes while preserving features necessary for desirable classification tasks.

As illustrated in Figure 3, while the encodings are visually noisy, they successfully achieve the target accuracy trade-offs. For instance, an encoding designed to inhibit "Army Base" but promote "Airport Terminal" results in a 65% accuracy for the former and 92% for the latter.

Conclusion

This work demonstrates that persistent privacy in image encodings is achievable through targeted adversarial losses and architectural constraints. By focusing on the removal of redundant cues and ensuring output diversity through per-location normalization, the framework provides a robust defense against high-capacity adversaries. Future work may focus on making these encodings more compatible with off-the-shelf pre-trained models by incorporating generative constraints to produce more "natural-looking" outputs.

Summary updated

Abstract

We present a framework to learn privacy-preserving encodings of images that inhibit inference of chosen private attributes, while allowing recovery of other desirable information. Rather than simply inhibiting a given fixed pre-trained estimator, our goal is that an estimator be unable to learn to accurately predict the private attributes even with knowledge of the encoding function. We use a natural adversarial optimization-based formulation for this---training the encoding function against a classifier for the private attribute, with both modeled as deep neural networks. The key contribution of our work is a stable and convergent optimization approach that is successful at learning an encoder with our desired properties---maintaining utility while inhibiting inference of private attributes, not just within the adversarial optimization, but also by classifiers that are trained after the encoder is fixed. We adopt a rigorous experimental protocol for verification wherein classifiers are trained exhaustively till saturation on the fixed encoders. We evaluate our approach on tasks of real-world complexity---learning high-dimensional encodings that inhibit detection of different scene categories---and find that it yields encoders that are resilient at maintaining privacy.
BibTeX
@inproceedings{pittaluga2018learning,
  author = {Francesco Pittaluga and Sanjeev J. Koppal and Ayan Chakrabarti},
  title = {{Learning Privacy Preserving Encodings through Adversarial Training}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  doi = {10.1109/wacv.2019.00089},
}
Figure