Blind Baselines Beat Membership Inference Attacks for Foundation Models

Debeshee Das, Jie Zhang, Florian TramèrSubmission to ICLR 2025 · Rejected

TL;DR

Current membership inference (MI) benchmarks for foundation models are fundamentally broken because member and non-member data are drawn from noticeably different distributions. The authors demonstrate that simple "blind" baselines—which never query the target model and instead classify membership using basic text heuristics like dates and n-grams—consistently outperform state-of-the-art MI attacks across nine published benchmarks.

What Problem Does This Paper Address?

Membership inference attacks aim to determine whether a specific data sample was included in a model's training set. In foundation models, MI is widely promoted as a tool to audit training privacy, trace copyrighted material, and verify machine unlearning.

Evaluating an MI attack typically requires comparing training samples (members) against held-out samples (non-members) drawn from the exact same distribution. However, because foundation models are trained on massive, often undisclosed web scrapes lacking designated test sets, researchers have relied on post-hoc dataset creation—for instance, selecting members published before a model's knowledge cutoff and non-members published after, or combining different web datasets.

This paper investigates whether these post-hoc datasets introduce severe distribution shifts that invalidate existing MI evaluations.

Methods

Rather than designing a complex model-auditing attack, the authors introduce intentionally naive blind attacks that classify membership entirely without inspecting the model's weights, activations, or loss values:

  • Date Detection: Regular expressions extract dates mentioned within a text snippet. If all referenced dates fall prior to a training cutoff, the sample is predicted to be a member.
  • Bag-of-Words Classifiers: Standard text classifiers trained on an 80/20 train/test split (evaluated using 10-fold cross-validation) to determine membership solely from word frequencies.
  • Greedy Rare $n$-gram Selection: A simple greedy search over character or word $n$-grams ($n \in [1, 5]$) ranked by their true-positive rate to false-positive rate (TPR/FPR) ratio. A held-out sample is classified as a member if it contains any of the selected $n$-grams.

Datasets

The authors evaluate their blind baselines against nine published MI evaluation benchmarks across text and vision-language modalities:

  • Temporal shift benchmarks: WikiMIA, BookMIA, Temporal Wiki, Temporal ArXiv, ArXiv (all vs. 1 month), and ArXiv (1 month vs. 1 month).
  • Biased replication benchmarks: Multi-Webdata (CC12M members vs. CC3M/LAION/MS-COCO non-members), LAION-MI (LAION members vs. translated Multilingual-LAION non-members), and Project Gutenberg (PG-19 members vs. post-2019 additions).

No new evaluation benchmarks were created; instead, the authors point to established pre-training datasets with official, random train/test splits—such as The Pile, DataComp, and DataComp-LM (DCLM)—as rigorous paths forward.

Key Insights and Ideas

  • Post-hoc debiasing is extremely brittle: Even benchmarks engineered explicitly to remove distribution shifts leave distinct statistical artifacts in the tails:
  • In LAION-MI, non-members were translated using Facebook's M2M100 model and carefully filtered to match aesthetic scores and feature PCA distributions. Despite this, non-breaking spaces (\xa0) and residual Cyrillic characters present in the English LAION scrape—but absent from the translation model's output vocabulary—readily distinguish members from non-members.
  • In Project Gutenberg, differences in metadata formatting (such as .htm versus .html URLs in book prefaces added after 2019) allowed trivial identification. Even ignoring metadata, raw text vocabulary differences achieved a 16.6% TPR at 1% FPR.
  • Temporal proximity does not prevent topic drift: In the arXiv benchmark narrowing members to February 2023 and non-members to March 2023, terms like "GPT-4" (released March 14, 2023) or "iccvfinalcopy" were ~25× more likely to appear in non-members.
  • MI attacks perform "worse than chance": When a model-agnostic blind baseline outperforms published state-of-the-art MI attacks, those attacks are not demonstrating membership leakage—they are merely functioning as inefficient estimators of underlying dataset shift.

Key Results

Across all nine evaluation datasets, the blind attacks surpass the best reported MI attacks:

Dataset Metric Prior Best MI Attack Blind Baseline (Ours) Baseline Method
WikiMIA TPR @ 5% FPR 43.2% 94.7% Bag-of-Words
BookMIA AUC ROC 88.0% 91.4% Bag-of-Words
Temporal Wiki AUC ROC 79.6% 79.9% Greedy $n$-gram
Temporal ArXiv AUC ROC 74.5% 75.3% Bag-of-Words
ArXiv (all vs. 1 mo) TPR @ 1% FPR 5.9% 10.6% Date Detection
ArXiv (1 mo vs. 1 mo) TPR @ 1% FPR 2.5% 2.7% Greedy Words
Multi-Webdata TPR @ 1% FPR 40.3% 93.0% Greedy $n$-gram
LAION-MI TPR @ 1% FPR 2.5% 8.9% Greedy $n$-gram
Gutenberg TPR @ 1% FPR 18.8% 55.1% Greedy $n$-gram

Limitations and Future Work

  • Scope of Blind Attacks: The proposed blind techniques are intentionally simple proofs-of-concept; they are not intended to serve as production-grade data classifiers.
  • Recommendation for the Field: Post-hoc MI evaluation datasets for closed foundation models should largely be abandoned for scientific auditing.
  • Future Direction: Future MI evaluations must rely on models trained with explicit, randomized independent and identically distributed (IID) train-test splits, such as open models trained on DataComp or DataComp-LM pools. When post-hoc sets are unavoidable, attacks must be evaluated strictly against strong blind baselines.

Summary updated

Abstract

Membership inference (MI) attacks try to determine if a data sample was used to train a machine learning model. For foundation models trained on unknown Web data, MI attacks are often used to detect copyrighted training materials, measure test set contamination, or audit machine unlearning. Unfortunately, we find that evaluations of MI attacks for foundation models are flawed, because they sample members and non-members from different distributions. For 9 published MI evaluation datasets, we show that blind attacks---that distinguish the member and non-member distributions without looking at any trained model---outperform state-of-the-art MI attacks. Existing evaluations thus tell us nothing about membership leakage of a foundation model's training data.
BibTeX
@misc{das2024blind,
  author = {Debeshee Das and Jie Zhang and Florian Tramèr},
  title = {{Blind Baselines Beat Membership Inference Attacks for Foundation Models}},
  year = {2024},
  eprint = {2406.16201},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.16201},
}
Figure