CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models

Jeffrey G. Wang, Jason Wang, Marvin Li, Seth Neel

TL;DR

Evaluating Membership Inference Attacks (MIAs) on Large Language Models has been notoriously flawed due to subtle distribution shifts in benchmark datasets that lead to false successes. By leveraging intermediate training checkpoints, CheckMIABench creates a truly clean, distribution-shift-free evaluation framework. The sobering result? Popular MIAs perform no better than random guessing when tested on properly controlled data.

What Problem Does This Paper Address?

Membership Inference Attacks (MIAs) aim to determine whether a specific piece of data was used to train a machine learning model. In the context of LLMs, highly accurate MIAs are crucial for auditing privacy leaks, verifying data copyright, and evaluating machine unlearning techniques.

However, evaluating MIAs on LLMs is incredibly difficult. Existing benchmarks often suffer from subtle distribution shifts between the "member" (trained) and "non-member" (untrained) datasets. For example, using Wikipedia articles from before 2017 as members and after 2023 as non-members introduces temporal shifts in writing styles. Previous work showed that a simple "blind" bag-of-words classifier, with absolutely zero access to the LLM, could distinguish members from non-members with over 94% accuracy simply by exploiting these dataset differences. This means prior MIA successes were largely an illusion caused by flawed benchmarks rather than actual privacy leakage from the models.

Methods

The authors introduce a clever, principled approach to eliminate distribution shifts by exploiting the mechanics of LLM pre-training.

During pre-training, data is tokenized, packed into fixed-size chunks, and randomly shuffled before being fed to the model. The authors realize that data processed before a specific training checkpoint and data processed after it are drawn from the exact same underlying distribution.

Figure 1: The shuffling of packed samples to construct the training data of Pythia and our checkpoint-based method to evaluate MIAs. In this scenario, we would sample from chunks {B2,B4,B1} for member data and chunk {B3} for the non-member data to evaluate MIAs for the model at Checkpoint 300.

To evaluate an MIA at checkpoint $T$:

  1. Data seen by the model before checkpoint $T$ is labeled as members.
  2. Data scheduled to be seen immediately after checkpoint $T$ is labeled as non-members.
  3. The MIA is evaluated on the model at state $T$.

To ensure their data splits are truly clean, the authors implement "Blind MIAs"—supervised classifiers (like Logistic Regression and Random Forests) trained on text features (Bag-of-Words, TF-IDF, Word2Vec, BERT) without model access. If these blind baselines achieve an Area Under the ROC Curve (AUC) of 0.5 (random guessing), it proves the dataset split is free of distribution shifts.

The authors have open-sourced their framework as a modular Python library called pandora_llm.

Datasets

The framework was instantiated and validated using two prominent open-source LLM families:

  • The Pile: Used to evaluate the Pythia model family across sizes ranging from 70M to 2.8B parameters.
  • Dolma: Used to evaluate the OLMo 7B parameter model.

Key Insights and Ideas

  • Shuffling is the Key to Clean Evaluation: The random shuffling of packed data chunks during pre-training provides a natural, mathematically sound way to generate perfectly independent and identically distributed (i.i.d.) member and non-member sets.
  • Current MIAs Don't Work on Pre-training Data: When the safety rails are properly put in place and distribution shifts are eliminated, existing state-of-the-art MIAs completely fail. They do not extract meaningful privacy signals from standard pre-trained language model checkpoints.
  • Deduplication Artifacts are Dangerous: The paper highlights how ex-post deduplication (cleaning the non-member set against the training set) alters the marginal distribution of the data, inadvertently creating a signal that blind classifiers can exploit, thus invalidating the benchmark.

Key Results

The authors first ran their "Blind MIA" sanity checks on their newly constructed splits for both The Pile and Dolma. As shown below, the blind classifiers achieved an AUC of ~0.5, confirming that the member and non-member sets are distributionally identical.

Table 1: Sanity check: supervised model-free (“blind") MIAs get no signal on our cleaned split of The Pile.

Next, they benchmarked a half-dozen popular MIAs from recent literature—including LOSS, Min-K, Min-K%++, Zlib, MoPe, and ReCaLL. Across all model sizes for Pythia and OLMo, none of the attacks achieved an AUC statistically higher than the 0.5 random baseline.

Table 2: We benchmark several MIAs against the Pythia family of models on our cleaned split of The Pile.

Even on the large OLMo 7B model, the attacks failed to achieve traction, with AUCs hovering tightly around 0.49 to 0.52.

Table 4: We benchmark several published MIAs against OLMo 7B on our cleaned split of Dolma.

Limitations and Future Work

While CheckMIABench provides a rigorous foundation, it has a few limitations:

  • Requires White-Box Training Info: The framework requires open-source models where the exact training data order and intermediate checkpoints are publicly available. It cannot be easily applied to closed-source models like GPT-4 or models with hidden training recipes like Llama or Qwen.
  • Scope of Data: The evaluation was limited to models trained almost exclusively on English web data and capped at a model size of 7B parameters.

Future work includes scaling these rigorous evaluations to larger models, different languages, and distinct data subsets (like source code) where memorization patterns might differ.

Summary updated

Abstract

Membership inference attacks (MIAs) are a canonical way to assess a machine learning model's privacy properties. Although several attempts have been made to evaluate MIAs on language models, the extant literature has suffered numerous difficulties in constructing clean evaluations to test new techniques. In particular, subtle distribution shifts between member and non-member sets can undermine the statistical validity of MIAs; recent work has underscored this by showing that "blind" methods with no access to the underlying model can perform far better than published methods on the same benchmarks. This paper constructs a benchmark for principled evaluation of MIAs against LLMs, by leveraging the insight that training data before and after a fixed point during training are drawn from the same distribution. Therefore, all open-source models with intermediate checkpoints and public training data can be converted into MIA testbeds. We apply our framework to a half-dozen published attacks on the Pythia and OLMo family of models, from 70M to 7B parameters. To facilitate further privacy research, we open-source a modular library for designing and implementing attacks in this setting: https://github.com/safr-ai-lab/pandora_llm.
BibTeX
@inproceedings{wang2026checkmiabench,
  author = {Jeffrey G. Wang and Jason Wang and Marvin Li and Seth Neel},
  title = {{CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.30},
}

Loading related content...

Figure