ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learning

Sasi Kumar Murakonda, Reza ShokriarXiv 2020 · arXiv

Introduction

Machine learning models are often perceived as abstract mathematical functions that generalize patterns from data. However, high-capacity models, such as deep neural networks, frequently "memorize" specific details of their training sets. This memorization creates a significant privacy vulnerability: an adversary may be able to determine whether a specific individual's data was used to train a model simply by analyzing the model's outputs or internal parameters.

ML Privacy Meter is an open-source Python library designed to address this challenge. It quantifies the extent to which a model "leaks" sensitive training information by simulating state-of-the-art membership inference attacks (MIAs). By providing a concrete, quantitative risk score, the tool assists organizations in meeting regulatory requirements, such as the General Data Protection Regulation (GDPR) mandate for Data Protection Impact Assessments (DPIA).

Methodology: Quantifying Indirect Leakage

The core functionality of ML Privacy Meter involves acting as a "white-hat" attacker to evaluate a model's privacy robustness. The tool supports two primary attack vectors:

  1. Black-box Access: Simulates an attacker with access only to model predictions (e.g., confidence scores), typical of "Machine Learning as a Service" (MLaaS) environments.
  2. White-box Access: Simulates an attacker with access to internal model parameters, relevant for federated learning or models deployed on untrusted infrastructure.
The ML Privacy Meter workflow: from training data to a comprehensive privacy risk report.

The tool calculates a "privacy risk score" for each data record, representing the attacker’s confidence that the record was part of the training set. This granular approach allows practitioners to identify specific high-risk records rather than relying on a single aggregate metric for the entire model.

Key Insights: Empirical Risk vs. Theoretical Guarantees

A critical contribution of this work is the distinction between theoretical privacy guarantees and empirical risk. While techniques like Differential Privacy (DP) provide a theoretical "worst-case" bound (the $\epsilon$ parameter), ML Privacy Meter measures the actual, practical risk encountered in a specific deployment. This empirical measurement allows developers to calibrate privacy parameters to balance utility and safety effectively.

Furthermore, the tool aligns directly with the three stages of a DPIA: identifying potential threats, assessing the impact on individuals, and selecting appropriate mitigation strategies, such as regularization or sub-sampling.

Experimental Results and Risk Assessment

The tool evaluates attack success using Receiver Operating Characteristic (ROC) curves. An attack performing no better than random chance yields an Area Under the Curve (AUC) of 0.5, while higher values indicate greater privacy leakage.

Quantitative risk assessment showing an ROC curve with an AUC of 0.81 and a membership probability distribution for a sample class.

In illustrative examples, the tool demonstrated an AUC of 0.81, signaling a significant privacy leak. The generated histograms visualize the separation between the membership probabilities of training data (members) and population data (non-members). A wider gap between these distributions indicates a higher susceptibility to membership inference.

Conclusion and Future Directions

ML Privacy Meter provides a standardized framework for measuring indirect data leakage, bridging the gap between academic research and regulatory compliance. While currently focused on membership inference, future iterations aim to encompass attribute inference and model inversion attacks. Integrating these assessments into the standard ML development lifecycle will be essential for automating the trade-off between model utility and data privacy.

Summary updated

Abstract

When building machine learning models using sensitive data, organizations should ensure that the data processed in such systems is adequately protected. For projects involving machine learning on personal data, Article 35 of the GDPR mandates it to perform a Data Protection Impact Assessment (DPIA). In addition to the threats of illegitimate access to data through security breaches, machine learning models pose an additional privacy risk to the data by indirectly revealing about it through the model predictions and parameters. Guidances released by the Information Commissioner's Office (UK) and the National Institute of Standards and Technology (US) emphasize on the threat to data from models and recommend organizations to account for and estimate these risks to comply with data protection regulations. Hence, there is an immediate need for a tool that can quantify the privacy risk to data from models. In this paper, we focus on this indirect leakage about training data from machine learning models. We present ML Privacy Meter, a tool that can quantify the privacy risk to data from models through state of the art membership inference attack techniques. We discuss how this tool can help practitioners in compliance with data protection regulations, when deploying machine learning models.
BibTeX
@misc{murakonda2020ml,
  author = {Sasi Kumar Murakonda and Reza Shokri},
  title = {{ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learning}},
  year = {2020},
  eprint = {2007.09339},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.09339},
}
Figure