Formalizing Distribution Inference Risks

Anshuman Suri, David EvansarXiv 2021 · OpenReview

TL;DR

This paper introduces a formal framework for "distribution inference attacks," a type of privacy attack that aims to infer statistical properties of the distribution from which a machine learning model's training data was sampled, rather than just properties of the specific training dataset itself. They demonstrate this framework can capture existing attacks and introduce new ones, revealing that even subtle shifts in data distributions can be detectable.

What Problem Does This Paper Address?

Machine learning models are trained on data, and this training process can inadvertently leak information about the data itself. While "membership inference attacks" (determining if a specific data point was in the training set) and "attribute inference attacks" (inferring specific attributes of training data) are well-studied, there's a gap in formally understanding and quantifying "property inference attacks." These attacks aim to uncover broader statistical characteristics of the underlying data distribution, which could be sensitive. For instance, knowing the proportion of a certain demographic in a medical dataset used to train a diagnostic model could be a privacy concern. Existing formalizations often focus on individual data points or specific attribute ratios, failing to capture a wider range of potential distribution-level properties.

Methods

The authors propose a formal, generic definition of property inference attacks based on a cryptographic game. In this game, a "trainer" randomly chooses one of two distribution-altering functions ($G_0$ or $G_1$) and samples a training dataset from the resulting distribution. They then train a model ($M$) on this dataset. An "adversary" observes the trained model $M$ and tries to infer which distribution function ($G_0$ or $G_1$) was used. If the adversary can reliably distinguish between the two, they have successfully performed a property inference attack.

This framework is flexible:

  • $G_0$ can represent the original data distribution.
  • $G_1$ can represent a modified distribution with a specific property (e.g., a different ratio of a certain attribute).
  • The "property" can be any characteristic of the distribution, not just simple attribute ratios.

They then implement and evaluate three attack strategies:

  1. Loss Test: Leverages the intuition that a model trained on a distribution will have lower test loss on data sampled from that same distribution.
  2. Threshold Test: Trains separate models on data from each potential distribution and uses a threshold on their performance to distinguish them.
  3. Meta-Classifiers: Employs state-of-the-art permutation-invariant networks that take model parameters (weights, biases) as input to classify which distribution was used.

Datasets

The paper uses three distinct datasets to test their framework and attacks:

  • Census: A tabular dataset for predicting income, used to infer properties related to the proportion of "whites" (race) and "females" (sex).
  • RSNA Bone Age: An image dataset for predicting bone age, where they focus on inferring the proportion of "females" (metadata).
  • ogbn-arxiv: A large graph dataset representing citations between arXiv papers, used to infer the "mean node-degree" property.

For each dataset, they carefully construct non-overlapping data splits for the adversary and victim, and for training/testing, ensuring that property inference is the goal, not direct dataset inference.

Key Insights and Ideas

  • Distribution vs. Dataset Inference: The core insight is that property inference attacks target the distribution from which data is sampled, a fundamentally different goal than inferring properties of specific training records. This distinction is crucial for understanding the privacy risks.
  • Generality of the Framework: The proposed formalization is remarkably general, capable of capturing not only simple attribute ratio inference but also more complex properties like graph structures (e.g., mean node degree).
  • Subtle Distribution Shifts are Detectable: Even small changes in the underlying data distribution can be enough for a sophisticated adversary to infer which distribution was used to train a model.
  • Intuition vs. Reality: The paper highlights that adversarial success doesn't always align with intuitive notions of how "divergent" two distributions are. For example, a distribution with 0% or 100% of a certain attribute might be harder to distinguish than one with 30% vs. 70%.
  • Meta-classifiers are Powerful but Resource-Intensive: State-of-the-art meta-classifier attacks show high accuracy, but require significant computational resources and access to many trained models.

Key Results

  • Census Dataset: The Threshold Test generally outperformed other methods, with meta-classifiers showing a clear trend of improved accuracy as the female proportion diverged from 0.5. However, the attack was less effective for the "white" proportion, with meta-classifiers barely above random guessing except at the extremes.
  • RSNA Bone Age Dataset: Meta-classifiers were highly effective at distinguishing female proportions, even when close to 0.5. The loss test also showed non-trivial accuracy, sometimes approaching meta-classifier performance.
  • ogbn-arxiv Dataset: For inferring mean node-degree, both Loss Test and Threshold Test failed. However, the meta-classifier was highly successful, even distinguishing between distributions with a difference of just one in mean node-degree. A regression variant of the meta-classifier could even predict the actual average node degree.
  • Attack Performance: The paper demonstrates that meta-classifiers can achieve accuracies ranging from around 60% to over 90% depending on the dataset and property. The Loss Test often performed slightly better than random guessing, while the Threshold Test showed variable but sometimes strong performance.

Limitations and Future Work

  • Realistic Settings: The authors note that in more realistic settings (e.g., ensuring no overlap between victim/adversary data, random sampling), meta-classifier performance can drop, suggesting that current attacks might be overly optimistic in some idealized scenarios.
  • Improving Simpler Attacks: The paper suggests there's room for improvement in attacks that don't rely on computationally expensive meta-classifiers.
  • Broader Property Inference: Future work could explore a wider range of distribution properties and attack scenarios, further understanding the privacy implications of machine learning models.
  • Defense Mechanisms: While not the focus, the work implicitly calls for the development of defenses against these distribution inference attacks.

Summary updated

Abstract

Property inference attacks reveal statistical properties about a training set but are difficult to distinguish from the primary purposes of statistical machine learning, which is to produce models that capture statistical properties about a distribution. Motivated by Yeom et al.'s membership inference framework, we propose a formal and generic definition of property inference attacks. The proposed notion describes attacks that can distinguish between possible training distributions, extending beyond previous property inference attacks that infer the ratio of a particular type of data in the training data set. In this paper, we show how our definition captures previous property inference attacks as well as a new attack that reveals the average degree of nodes of a training graph and report on experiments giving insight into the potential risks of property inference attacks.
BibTeX
@misc{suri2021formalizing,
  author = {Anshuman Suri and David Evans},
  title = {{Formalizing Distribution Inference Risks}},
  year = {2021},
  eprint = {2106.03699},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.03699},
}
Figure