Rethinking Privacy Preserving Deep Learning: How to Evaluate and Thwart Privacy Attacks

Lixin Fan, Kam Woh Ng, Ce Ju, Tianyu Zhang, Chang Liu, Chee Seng Chan, Qiang Yang

TL;DR

This paper introduces a new framework for evaluating the trade-off between model utility and data privacy in deep learning. The authors formulate privacy attacks as solving systems of linear equations and propose the Secret Polarization Network (SPN)—a novel defense that uses private, adaptive gradient perturbations to thwart attacks while actually improving model accuracy by 5-20% compared to traditional differential privacy methods.

What Problem Does This Paper Address?

In Privacy-Preserving Deep Learning (PPDL), particularly in Federated Learning, multiple parties train a model together without sharing their raw data. Instead, they share "gradients" (updates to the model). However, recent research has shown a terrifying reality: an adversary can often "reconstruct" the original private images just by looking at these shared gradients.

The standard defense is Differential Privacy (DP), which adds random noise to the gradients. The problem? Too much noise makes the model inaccurate (low utility), while too little noise leaves the data vulnerable. There hasn't been a standard, objective way to measure this "utility-vs-privacy" trade-off, and existing defenses often fail to stop sophisticated reconstruction attacks without destroying model performance.

Methods

The authors attack the problem from three angles:

  1. New Evaluation Metrics: They propose Privacy-Preserving Characteristic (PPC) curves, which plot model accuracy against privacy loss (the distance between reconstructed and original data). To summarize this curve into one number, they introduce Calibrated Averaged Performance (CAP)—essentially the "Area Under the Curve" for privacy-utility balance.
  2. Theoretical Attack Formulation: They prove that reconstruction attacks are essentially attempts to solve a noisy system of linear equations ($BI \cdot x = WI$). By analyzing this, they identify the mathematical condition under which an attack is guaranteed to fail.
  3. Secret Polarization Network (SPN): This is their flagship defense. Unlike DP, which adds "blind" random noise, SPN uses a dual-headed architecture:
  4. Public Backbone: Shared with everyone.
  5. Private SPN Head: Kept secret by the participant.
  6. The SPN head adds "secret and element-wise adaptive gradients" to the shared updates. Because the attacker doesn't know the SPN parameters, they can't "subtract out" this perturbation. Crucially, the SPN uses a "polarization loss" that acts as a regularizer, helping the model learn better features while simultaneously confusing the attacker.

Datasets

The researchers evaluated their methods on four standard computer vision benchmarks:

  • MNIST: Handwritten digits.
  • CIFAR-10 & CIFAR-100: Small color images across 10 or 100 classes.
  • SVHN: Street View House Numbers.

They used these to test three types of threats: reconstruction attacks (recovering the image), membership attacks (determining if a specific image was in the training set), and tracing attacks (identifying which participant provided a specific data item).

Key Insights and Ideas

  • Privacy as a Linear System: The "aha moment" here is treating deep leakage as a math problem. By showing that reconstruction is a linear inversion, they provide a theoretical "safety zone" where noise is mathematically guaranteed to defeat the attacker.
  • Noise Can Be Useful: Traditionally, noise is seen as a "tax" on accuracy. SPN flips this. Because the "noise" is actually a structured gradient from a secondary task (polarization), it helps the model generalize better. It's like adding a secret ingredient that both hides the recipe and makes the food taste better.
  • The Failure of Simple Pruning: The paper confirms that simply "pruning" gradients (sharing only the largest ones) is not a reliable defense, as attackers can still reconstruct data from the remaining fragments.

Key Results

  • Accuracy Boost: In regimes where data privacy was satisfactorily protected, SPN improved model accuracies by 5-20% compared to baseline differential privacy and gradient pruning methods.
  • Federated Learning Gains: When integrated with the standard FedAvg algorithm, SPN consistently improved global model accuracy by 2-3% for CIFAR-10 and about 0.2% for MNIST compared to standard training.
  • Robustness: SPN showed a "pronounced improvement" in CAP values across all attack types (reconstruction, membership, and tracing), meaning it offers a much better "bang for your buck" regarding the privacy-utility trade-off.

Limitations and Future Work

  • Private Parameters: The SPN defense relies on participants keeping a small portion of their network private. While this is feasible in many federated settings, it adds a layer of management complexity.
  • Adversarial Examples: The authors note that while SPN protects against privacy attacks, the "adversarial" nature of the polarization might make the model vulnerable to adversarial example attacks (input perturbations designed to cause misclassification).
  • Unified Framework: The authors suggest that future work should focus on a unified framework that thwarts both privacy attacks and adversarial example attacks simultaneously.

Summary updated

Abstract

This chapter investigates capabilities of Privacy-Preserving Deep Learning (PPDL) mechanisms against various forms of privacy attacks. First, we propose to quantitatively measure the trade-off between model accuracy and privacy losses incurred by reconstruction, tracing and membership attacks. Second, a novel Secret Polarization Network (SPN) is proposed to thwart privacy attacks, which is highly competitive against existing PPDL methods. Extensive experiments showed that model accuracies are improved on average by 5–20% compared with baseline mechanisms, in regimes where data privacy are satisfactorily protected.
BibTeX
@inproceedings{fan2020rethinking,
  author = {Lixin Fan and Kam Woh Ng and Ce Ju and Tianyu Zhang and Chang Liu and Chee Seng Chan and Qiang Yang},
  title = {{Rethinking Privacy Preserving Deep Learning: How to Evaluate and Thwart Privacy Attacks}},
  booktitle = {Federated Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=lweU9MjOXbB},
}
Figure