Provable trade-offs between private & robust machine learning

Jamie HayesarXiv 2020 · OpenReview

TL;DR

This paper investigates the relationship between adversarial robustness and membership inference privacy in machine learning models. It demonstrates that, contrary to common belief, adversarial training doesn't always decrease privacy; in some settings, robust models can actually be more private than standard ones, depending on the training data size and the strength of adversarial perturbations.

What Problem Does This Paper Address?

Machine learning models are increasingly deployed in sensitive applications, raising concerns about their security and privacy. While significant research has focused on making models robust to adversarial examples (inputs subtly modified to fool the model), this often comes at the cost of increased overfitting. Overfitting means the model has learned the training data too well, making it easier for attackers to infer whether a specific data point was part of the training set (membership inference attack), thus breaching privacy. The prevailing assumption has been that achieving adversarial robustness inherently compromises privacy. This paper challenges that assumption by exploring the nuanced interplay between these two properties.

Methods

The authors analyze the "loss gap" – the difference between a model's training loss and its test loss – as a proxy for overfitting and, consequently, privacy risk. They develop theoretical frameworks to characterize how adversarial training, specifically with varying perturbation sizes ($\epsilon$) and training set sizes ($n$), affects this loss gap. They prove that adversarial training can either increase or decrease the loss gap compared to standard training, depending on the specific parameters. This theoretical analysis is then validated through empirical experiments on image classification datasets.

Datasets

  • CIFAR-10: Used for empirical validation of the theoretical findings.
  • CIFAR-100: Also used for empirical validation.
  • Gaussian Data: A simplified synthetic dataset used for theoretical analysis and initial empirical validation of the proposed relationships.

Key Insights and Ideas

  • The Privacy-Robustness Trade-off is Nuanced: It's not a universal rule that robust models are less private. The relationship is highly dependent on specific training conditions.
  • Training Set Size is Crucial: The amount of training data significantly influences whether adversarial training improves or degrades privacy. With more data, robust models can become more private.
  • Perturbation Strength Matters: The magnitude of adversarial perturbations ($\epsilon$) used during training plays a critical role in determining the privacy outcome.
  • Loss Gap as a Privacy Indicator: The paper solidifies the connection between overfitting (measured by the loss gap) and membership inference vulnerability. A smaller loss gap implies less overfitting and thus better privacy.

Key Results

  • Theoretical Proofs: The paper provides theoretical guarantees showing that adversarial training can lead to provably smaller loss gaps (and thus better privacy) than standard training under certain conditions (e.g., when $\epsilon$ is sufficiently large relative to the data characteristics). Conversely, it also shows settings where robust models are less private.
  • Empirical Validation: Experiments on CIFAR-10 and CIFAR-100 datasets confirm the theoretical predictions. For instance, with a small perturbation size ($\epsilon = 8/255$) and moderate training set sizes (200-500 samples), robust models exhibited better privacy than standard models.
  • Data Size Impact: The figures and analysis demonstrate that as the training set size increases, the loss gap (and thus privacy risk) generally decreases for both standard and robust models, but robust models can maintain a smaller loss gap for larger datasets.

Limitations and Future Work

  • Gaussian Data Assumption: The theoretical analysis is primarily conducted on a simplified Gaussian data model. While empirical results on image datasets support these findings, extending the theoretical framework to more complex data distributions remains an open area.
  • Membership Attack Strength: The authors acknowledge that current membership inference attacks are not perfect and might not fully reveal the correspondence between the loss gap and privacy vulnerability. Developing stronger attacks could provide a more definitive understanding.
  • Practical Implications: While the paper shows that robust models can be more private, further research is needed to translate these findings into practical, robust privacy-preserving machine learning systems.
  • Fairness: The paper focuses on privacy and robustness, but does not delve into the impact on fairness, which is another crucial aspect of secure and safe ML.

Summary updated

Abstract

Historically, machine learning methods have not been designed with security in mind. In turn, this has given rise to adversarial examples, carefully perturbed input samples aimed to mislead detection at test time, which have been applied to attack spam and malware classification, and more recently to attack image classification. Consequently, an abundance of research has been devoted to designing machine learning methods that are robust to adversarial examples. Unfortunately, there are desiderata besides robustness that a secure and safe machine learning model must satisfy, such as fairness and privacy. Recent work by Song et al. (2019) has shown, empirically, that there exists a trade-off between robust and private machine learning models. Models designed to be robust to adversarial examples often overfit on training data to a larger extent than standard (non-robust) models. If a dataset contains private information, then any statistical test that separates training and test data by observing a model's outputs can represent a privacy breach, and if a model overfits on training data, these statistical tests become easier. In this work, we identify settings where standard models will overfit to a larger extent in comparison to robust models, and as empirically observed in previous works, settings where the opposite behavior occurs. Thus, it is not necessarily the case that privacy must be sacrificed to achieve robustness. The degree of overfitting naturally depends on the amount of data available for training. We go on to characterize how the training set size factors into the privacy risks exposed by training a robust model on a simple Gaussian data task, and show empirically that our findings hold on image classification benchmark datasets, such as CIFAR-10 and CIFAR-100.
BibTeX
@misc{hayes2020provable,
  author = {Jamie Hayes},
  title = {{Provable trade-offs between private \& robust machine learning}},
  year = {2020},
  eprint = {2006.04622},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.04622},
}
Figure