Blind Baselines Beat Membership Inference Attacks for Foundation Models

Debeshee Das, Jie Zhang, Florian Tramèr

TL;DR

Membership inference (MI) attacks aim to determine if a data point was used to train a machine learning model, with applications in privacy and copyright detection for foundation models. This paper reveals that current MI evaluation datasets are fundamentally flawed, exhibiting significant distribution shifts between member and non-member data. As a result, simple "blind" attacks that don't even look at the model consistently outperform state-of-the-art MI attacks, rendering current evaluations unreliable.

What Problem Does This Paper Address?

The paper tackles a critical issue in evaluating the security and privacy of large foundation models: the reliability of membership inference (MI) attacks. MI attacks are designed to infer whether a specific data sample was part of a model's training set. This capability is important for detecting copyrighted material in training data, assessing test set contamination, and verifying the effectiveness of "unlearning" techniques.

However, evaluating MI attacks on foundation models is challenging because their training data is often unknown and not publicly available. To overcome this, researchers have created custom evaluation datasets by sampling "members" from suspected training sources and "non-members" from other sources. The problem, as highlighted by this paper, is that these methods of creating member and non-member sets are deeply flawed. They inadvertently introduce significant distribution shifts between the two groups, making them easily distinguishable without even needing to query the target model. This means that current MI attack evaluations are not truly measuring the model's ability to leak membership information but rather its ability to exploit these artificial data biases.

Methods

The core contribution of this paper is the demonstration that simple, "blind" attacks can outperform sophisticated MI attacks on existing foundation model evaluation datasets. These blind attacks do not interact with or learn from the target model at all. Instead, they exploit the distribution shifts present in the datasets themselves. The authors propose several such blind attack techniques:

  • Date Detection: For datasets with temporal components (e.g., data from Wikipedia or arXiv), they extract dates from text samples and use them to infer membership based on whether the dates fall before or after a certain threshold.
  • Bag-of-Words (BoW) Classification: They train simple BoW or n-gram classifiers on textual features of the data. These classifiers learn to distinguish members from non-members based on word frequencies, without any knowledge of the target model.
  • Greedy Rare Word Selection: A more refined approach where they greedily select n-grams (sequences of words) that have the highest ratio of true positive rate (TPR) to false positive rate (FPR) on a portion of the data. These selected n-grams are then used to classify new samples.

The key idea is to show that if a simple heuristic based on data characteristics can achieve high performance, then the MI attacks claiming to exploit model leakage are likely just exploiting these data biases.

Datasets

The paper evaluates its blind attacks on 8 publicly available MI evaluation datasets for foundation models, including both text and vision-text modalities. These datasets are:

  • WikiMIA
  • BookMIA
  • Temporal Wiki
  • Temporal ArXiv
  • ArXiv (all vs 1 month)
  • ArXiv (1 month vs 1 month)
  • LAION-MI
  • Gutenberg

The authors note that some datasets, like ArXiv and Gutenberg, were not publicly available and were replicated following the original collection steps. They also discuss the potential of datasets like The Pile, DataComp, and DataComp-LM as more promising benchmarks for future MI research due to their focus on random train-test splits.

Key Insights and Ideas

  • Current MI evaluations are fundamentally flawed: The most striking insight is that existing datasets used to evaluate MI attacks on foundation models are not designed to isolate model leakage. Instead, they contain inherent distribution shifts that make members and non-members easily distinguishable by simple, model-agnostic methods.
  • Blind attacks outperform state-of-the-art MI attacks: This is the paper's central claim and a major finding. Simple heuristics based on data features (like dates or word frequencies) achieve significantly better performance than MI attacks that are designed to exploit model behavior.
  • Membership leakage is not reliably measured: Because blind attacks perform so well, the paper argues that current MI evaluations tell us "nothing about membership leakage of a foundation model's training data." The reported successes of MI attacks may be artifacts of dataset construction rather than genuine model vulnerabilities.
  • Post-hoc dataset creation is brittle: The paper demonstrates that even when researchers attempt to create comparable member and non-member sets after the fact, subtle biases (temporal, linguistic, or formatting) can persist and be exploited by simple methods.

Key Results

The paper presents compelling quantitative results across the 8 evaluated datasets. In virtually all cases, their blind attacks significantly outperform the best reported MI attacks. For example:

  • On WikiMIA, a blind bag-of-words classifier achieved a near-perfect 94.7% TPR at 5% FPR, compared to the best reported attack's 43.2%.
  • For Gutenberg, a greedy n-gram attack achieved 55.1% TPR at 1% FPR, which is 3 times higher than the best MI attack.
  • Across all datasets, the blind attacks consistently achieved much higher metrics (TPR at low FPR or AUC ROC) than the best prior MI attacks, often by a substantial margin. Table 1 and Table 2 in the paper provide detailed comparisons.

Limitations and Future Work

Limitations:

  • Focus on Text Modality: While some datasets are multimodal, the paper primarily focuses on text-based blind attacks for simplicity and interpretability.
  • Dataset Specificity: The blind attacks are tailored to the specific biases found in each dataset. While this highlights the flaws, it doesn't offer a universal blind attack.
  • "Worse Than Chance" is Relative: The paper states that existing attacks perform "worse than chance" in the context of these flawed datasets, meaning their performance is lower than what a simple blind attack can achieve. This doesn't necessarily mean they are performing below random guessing on a truly unbiased dataset.

Future Work:

  • Rigorous Evaluation Benchmarks: The authors advocate for future MI evaluations to use datasets with truly random train-test splits, such as The Pile, DataComp, or DataComp-LM, to ensure that MI attacks are evaluated on their ability to detect actual model leakage, not data biases.
  • Stronger Blind Baselines: Future MI attacks should be compared against these stronger, model-agnostic blind baselines to demonstrate genuine progress.
  • Development of IID Splits: Future dataset and model developers should release official IID (independent and identically distributed) splits to enable more rigorous MI evaluations across various modalities.

Summary updated

Abstract

Membership inference (MI) attacks try to determine if a data sample was used to train a machine learning model. For foundation models trained on unknown Web data, MI attacks can be used to detect copyrighted training materials, measure test set contamination, or audit machine unlearning. Unfortunately, we find that evaluations of MI attacks for foundation models are flawed, because they sample members and non-members from different distributions. For 8 published MI evaluation datasets, we show that blind attacks -- that distinguish the member and non-member distributions without looking at any trained model -- outperform state-of-the-art MI attacks. Existing evaluations thus tell us nothing about membership leakage of a foundation model's training data.
BibTeX
@misc{das2024blind,
  author = {Debeshee Das and Jie Zhang and Florian Tramèr},
  title = {{Blind Baselines Beat Membership Inference Attacks for Foundation Models}},
  year = {2024},
  eprint = {2406.16201},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.16201},
}

Loading related content...

Figure