Noisy Neighbors: Efficient membership inference attacks against LLMs

Filippo Galli, Luca Melis, Tommaso CucinottaACL 2024

TL;DR

This paper introduces "noisy neighbors," an efficient method for membership inference attacks (MIAs) against large language models (LLMs). By injecting carefully crafted noise into the embedding space of a target sample, this approach achieves comparable privacy auditing effectiveness to more computationally expensive methods, requiring only inference-mode access to the LLM.

What Problem Does This Paper Address?

Large language models (LLMs) are increasingly powerful but raise significant privacy concerns due to their training on vast datasets that may contain sensitive personal information. Regulations like GDPR and CCPA necessitate robust auditing tools to assess these privacy risks. Membership Inference Attacks (MIAs) are a primary method for this, aiming to determine if a specific data point was used during an LLM's training. Traditional MIA approaches often require training auxiliary "shadow models" that mimic the target LLM, which is computationally demanding and time-consuming. This paper addresses the need for a more efficient and practical MIA method that can be employed by privacy auditors without extensive computational resources or deep knowledge of the target model's architecture.

Methods

The core innovation is the "noisy neighbors" strategy. Instead of training shadow models, this method artificially creates "noisy neighbors" of a target sample by injecting Gaussian noise directly into the sample's representation in the embedding space. Specifically, if $f(x) = g(e(x))$ represents an LLM where $e(x)$ is the embedding layer and $g$ is the rest of the network, the noisy neighbor is generated as $f(x) = g(e(x) + \rho)$, where $\rho$ is random noise drawn from a Gaussian distribution $N(0, \sigma I_n)$.

This noise injection shifts the comparison from evaluating the perplexity of different models on one sample to comparing the perplexity of the same model on different samples (the original and its noisy counterpart). The intuition is that a training sample will exhibit a smaller perplexity difference with its noisy neighbor compared to a non-training sample. The standard deviation $\sigma$ of the noise is a crucial parameter, and the paper shows its performance peaks at an optimal value, which can be efficiently found using binary search. This approach requires only query access to the target LLM in inference mode, sidestepping the need for shadow model training.

Datasets

The experiments were conducted using the GPT-2 small architecture, pre-trained on Open-WebText. For fine-tuning and evaluation, the WikiText corpus was used. The data was split, with 60% of WikiText used for fine-tuning the model and training shadow models (for comparison), and the remaining 40% used as a source for non-member samples.

Key Insights and Ideas

  • Efficiency through Noise: The paper demonstrates that introducing controlled noise into the embedding space is a computationally efficient way to achieve the same goal as training complex shadow models for MIA calibration.
  • Auditor-Centric Approach: The method is framed from the perspective of a privacy auditor, prioritizing practicality and reduced computational burden over developing a more potent attack.
  • Inference-Only Access: A significant advantage is that the attack only requires the target LLM to be in inference mode, making it applicable in scenarios where training or white-box access is not feasible.
  • Optimal Noise Level: The performance of the noisy neighbor attack is sensitive to the noise standard deviation ($\sigma$), with a clear optimal value that can be identified, suggesting a well-defined parameter tuning process.

Key Results

The "noisy neighbors" strategy showed remarkable alignment with the performance of shadow model-based MIAs.

  • The Area Under the Curve (AUC) for the ROC curve of the MIA classifier was comparable: 0.727 for the noisy strategy versus 0.753 for the shadow model strategy, a discrepancy of only ~3.4%.
  • At low False Positive Rates (FPRs), which are critical for confidently identifying members, the noisy strategy showed a strong overlap with the shadow strategy, indicating its effectiveness in distinguishing members from non-members.
  • The paper also reports empirical differential privacy measurements, showing strong consistency between the noisy and shadow strategies, particularly for FPRs below $10^{-2}$.

Limitations and Future Work

The authors acknowledge several limitations:

  • Generalizability: The effectiveness of the noisy neighbors method relies on assumptions about the model and dataset that may not universally apply.
  • Parameter Sensitivity: The success of the attack depends on specific noise parameters, potentially limiting its generalizability without careful tuning.
  • Computational Resources: While more efficient than shadow model training, the method still requires significant computational resources for generating multiple noisy neighbors and performing LLM inferences.

Future work could explore:

  • Investigating the applicability of this method to a wider range of LLM architectures and datasets.
  • Developing more automated or robust methods for determining optimal noise parameters.
  • Exploring the trade-offs between computational cost and attack effectiveness more deeply.

Summary updated

BibTeX
@inproceedings{galli2024noisy,
  author = {Filippo Galli and Luca Melis and Tommaso Cucinotta},
  title = {{Noisy Neighbors: Efficient membership inference attacks against LLMs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure