Statistical MIA: Rethinking Membership Inference Attack for Reliable Unlearning Auditing

Jialong Sun, Zeming Wei, Jiaxuan Zou, Jiacheng Gong, Guanheng Wang, Chengyang Dong, Jialong Li, Bo LiuarXiv 2026 · DOI

TL;DR

This paper introduces Statistical Membership Inference (SMI), a novel, training-free auditing method for machine unlearning. SMI addresses the fundamental flaw in existing Membership Inference Attacks (MIAs) that leads to overly optimistic unlearning evaluations, offering a more principled and efficient way to verify if a model has truly forgotten specific data.

What Problem Does This Paper Address?

Machine unlearning (MU) is crucial for privacy-sensitive applications, allowing users to exercise their "right to be forgotten." A key challenge is reliably auditing whether a model has truly forgotten specific training data. Current methods often rely on Membership Inference Attacks (MIAs), which try to determine if a data point was part of the training set. The assumption is that if an MIA fails to identify a sample as a member, the model has successfully forgotten it. However, this paper argues this assumption is fundamentally flawed. Unlearned samples can exhibit distinct feature distributions from genuine non-members, creating an "alignment bias" that leads to systematically optimistic evaluations of unlearning performance. Furthermore, training shadow models for MIAs is computationally expensive.

Methods

The authors propose Statistical Membership Inference (SMI), a training-free auditing framework. Instead of training a classifier (like traditional MIAs), SMI reformulates auditing as estimating the proportion of "non-member-like" features in the unlearned model's output distribution. This is framed as a mixture proportion estimation problem.

Two variants are introduced:

  • SMI: This method uses low-order moments (mean and covariance) of the member, non-member, and unlearned feature distributions to estimate the mixture proportion. It involves solving a constrained optimization problem derived from matching these moments.
  • SMI-M: To address potential instability in covariance estimation with limited samples, SMI-M uses kernel mean embeddings and Maximum Mean Discrepancy (MMD) to compare distributions. This transforms the problem into a convex quadratic programming problem, offering improved stability.

Both SMI and SMI-M avoid the need for training shadow models, significantly reducing computational overhead. SMI also provides bootstrap reference ranges to quantify the reliability of the auditing results.

Datasets

The experiments were conducted on image-classification tasks using the following datasets:

  • CIFAR-10
  • CIFAR-100
  • CINIC-10

Key Insights and Ideas

  • Failed MIA ≠ True Forgetting: The core insight is that a failed membership inference attack does not necessarily mean a model has forgotten a data point. Unlearned samples can occupy a distinct feature space, making them hard for MIAs to classify as non-members, but this doesn't equate to true forgetting.
  • Alignment Bias in MIAs: Conventional MIAs assume that unlearned sample features are close to non-member features. This assumption is fundamentally misaligned with the unlearning objective, where unlearned samples induce negative gradients, distinct from non-members.
  • Feature Space is Key: The authors shift the focus from parameter space to feature space for auditing, viewing unlearning as moving the unlearned feature distribution away from member features and towards non-member features.
  • Statistical Mixture Estimation: SMI offers a principled statistical approach by estimating the mixture proportion of member and non-member distributions in the unlearned features, providing a direct measure of forgetting.
  • Training-Free and Efficient: A major advantage of SMI is its training-free nature, eliminating the substantial computational cost associated with training shadow models for MIAs.

Key Results

  • Superior Performance: SMI-M consistently outperformed all MIA-based baselines across various datasets and unlearning settings, achieving better estimation accuracy for the forgetting rate.
  • Computational Efficiency: SMI methods have significantly lower computational costs compared to MIA methods that require training shadow models. For instance, training a single shadow model for a baseline method took orders of magnitude longer than running SMI-M.
  • Reliable Auditing: SMI demonstrated stable performance even with limited audit samples, converging around 200 samples or 20 samples per class. The bootstrap procedure for estimating uncertainty also showed minimal fluctuations.
  • Robustness: SMI-M proved robust across different kernel choices and bandwidth settings, with the RBF kernel and median bandwidth generally performing best.

Limitations and Future Work

  • Small Sample Sensitivity: SMI, being a statistical method, relies on a sufficient number of samples. Its performance on a single sample is weaker than MIAs, and at least 20 samples per class are suggested for effective auditing.
  • Adversarial Attacks: The paper notes that SMI can fail under adversarial perturbations designed to forge the unlearning process, leading to estimated mixture ratios greater than 1. However, such anomalous values can still indicate a problem.
  • Future Work: The authors suggest exploring Wasserstein methods for mixture proportion estimation, although their initial experiments showed SMI to be superior. They also aim to further analyze AI interpretability and the behavior of neural networks in feature space.

Summary updated

BibTeX
@misc{sun2026statistical,
  author = {Jialong Sun and Zeming Wei and Jiaxuan Zou and Jiacheng Gong and Guanheng Wang and Chengyang Dong and Jialong Li and Bo Liu},
  title = {{Statistical MIA: Rethinking Membership Inference Attack for Reliable Unlearning Auditing}},
  year = {2026},
  eprint = {2602.01150},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.01150},
}
Figure