TL;DR
Current membership inference (MI) benchmarks for foundation models are fundamentally broken because member and non-member data are drawn from noticeably different distributions. The authors demonstrate that simple "blind" baselines—which never query the target model and instead classify membership using basic text heuristics like dates and n-grams—consistently outperform state-of-the-art MI attacks across nine published benchmarks.
What Problem Does This Paper Address?
Membership inference attacks aim to determine whether a specific data sample was included in a model's training set. In foundation models, MI is widely promoted as a tool to audit training privacy, trace copyrighted material, and verify machine unlearning.
Evaluating an MI attack typically requires comparing training samples (members) against held-out samples (non-members) drawn from the exact same distribution. However, because foundation models are trained on massive, often undisclosed web scrapes lacking designated test sets, researchers have relied on post-hoc dataset creation—for instance, selecting members published before a model's knowledge cutoff and non-members published after, or combining different web datasets.
This paper investigates whether these post-hoc datasets introduce severe distribution shifts that invalidate existing MI evaluations.
Methods
Rather than designing a complex model-auditing attack, the authors introduce intentionally naive blind attacks that classify membership entirely without inspecting the model's weights, activations, or loss values:
- Date Detection: Regular expressions extract dates mentioned within a text snippet. If all referenced dates fall prior to a training cutoff, the sample is predicted to be a member.
- Bag-of-Words Classifiers: Standard text classifiers trained on an 80/20 train/test split (evaluated using 10-fold cross-validation) to determine membership solely from word frequencies.
- Greedy Rare $n$-gram Selection: A simple greedy search over character or word $n$-grams ($n \in [1, 5]$) ranked by their true-positive rate to false-positive rate (TPR/FPR) ratio. A held-out sample is classified as a member if it contains any of the selected $n$-grams.
Datasets
The authors evaluate their blind baselines against nine published MI evaluation benchmarks across text and vision-language modalities:
- Temporal shift benchmarks: WikiMIA, BookMIA, Temporal Wiki, Temporal ArXiv, ArXiv (all vs. 1 month), and ArXiv (1 month vs. 1 month).
- Biased replication benchmarks: Multi-Webdata (CC12M members vs. CC3M/LAION/MS-COCO non-members), LAION-MI (LAION members vs. translated Multilingual-LAION non-members), and Project Gutenberg (PG-19 members vs. post-2019 additions).
No new evaluation benchmarks were created; instead, the authors point to established pre-training datasets with official, random train/test splits—such as The Pile, DataComp, and DataComp-LM (DCLM)—as rigorous paths forward.
Key Insights and Ideas
- Post-hoc debiasing is extremely brittle: Even benchmarks engineered explicitly to remove distribution shifts leave distinct statistical artifacts in the tails:
- In LAION-MI, non-members were translated using Facebook's M2M100 model and carefully filtered to match aesthetic scores and feature PCA distributions. Despite this, non-breaking spaces (
\xa0) and residual Cyrillic characters present in the English LAION scrape—but absent from the translation model's output vocabulary—readily distinguish members from non-members. - In Project Gutenberg, differences in metadata formatting (such as
.htmversus.htmlURLs in book prefaces added after 2019) allowed trivial identification. Even ignoring metadata, raw text vocabulary differences achieved a 16.6% TPR at 1% FPR. - Temporal proximity does not prevent topic drift: In the arXiv benchmark narrowing members to February 2023 and non-members to March 2023, terms like
"GPT-4"(released March 14, 2023) or"iccvfinalcopy"were ~25× more likely to appear in non-members. - MI attacks perform "worse than chance": When a model-agnostic blind baseline outperforms published state-of-the-art MI attacks, those attacks are not demonstrating membership leakage—they are merely functioning as inefficient estimators of underlying dataset shift.
Key Results
Across all nine evaluation datasets, the blind attacks surpass the best reported MI attacks:
| Dataset | Metric | Prior Best MI Attack | Blind Baseline (Ours) | Baseline Method |
|---|---|---|---|---|
| WikiMIA | TPR @ 5% FPR | 43.2% | 94.7% | Bag-of-Words |
| BookMIA | AUC ROC | 88.0% | 91.4% | Bag-of-Words |
| Temporal Wiki | AUC ROC | 79.6% | 79.9% | Greedy $n$-gram |
| Temporal ArXiv | AUC ROC | 74.5% | 75.3% | Bag-of-Words |
| ArXiv (all vs. 1 mo) | TPR @ 1% FPR | 5.9% | 10.6% | Date Detection |
| ArXiv (1 mo vs. 1 mo) | TPR @ 1% FPR | 2.5% | 2.7% | Greedy Words |
| Multi-Webdata | TPR @ 1% FPR | 40.3% | 93.0% | Greedy $n$-gram |
| LAION-MI | TPR @ 1% FPR | 2.5% | 8.9% | Greedy $n$-gram |
| Gutenberg | TPR @ 1% FPR | 18.8% | 55.1% | Greedy $n$-gram |
Limitations and Future Work
- Scope of Blind Attacks: The proposed blind techniques are intentionally simple proofs-of-concept; they are not intended to serve as production-grade data classifiers.
- Recommendation for the Field: Post-hoc MI evaluation datasets for closed foundation models should largely be abandoned for scientific auditing.
- Future Direction: Future MI evaluations must rely on models trained with explicit, randomized independent and identically distributed (IID) train-test splits, such as open models trained on DataComp or DataComp-LM pools. When post-hoc sets are unavoidable, attacks must be evaluated strictly against strong blind baselines.
Summary updated