Mendata: A Framework to Purify Manipulated Training Data

Zonghao Huang, Neil Gong, Michael K. ReiterarXiv 2023 · arXiv · OpenReview

TL;DR

This paper introduces Mendata, a novel framework for purifying manipulated training data. Mendata perturbs untrusted data to match the distribution of a clean reference dataset, effectively removing hidden properties while preserving data utility. It achieves this through a min-max optimization approach and demonstrates strong performance against state-of-the-art data poisoning and data tracing attacks.

What Problem Does This Paper Address?

In machine learning, training data often comes from untrusted sources, making it vulnerable to manipulation. Data contributors might intentionally alter this data to embed hidden properties into the trained model. These properties can be exploited for malicious purposes, such as data poisoning attacks where the model is tricked into misclassifying specific inputs, or for data tracing, where the contributor can detect if their data was used without authorization. Existing defenses like prevention (modifying training algorithms) or detection (discarding suspicious data) have limitations, either sacrificing utility or being too restrictive. This paper addresses the need for a data purification method that can effectively remove these hidden properties without significantly degrading model accuracy.

Methods

Mendata tackles data purification by formulating it as an optimization problem. The core idea is to perturb the "untrusted" data points (those not in the clean reference set) so that their feature distributions become similar to the reference data. This similarity is quantified using the Wasserstein distance, a robust measure of distribution difference.

The optimization problem is a min-max formulation:

  1. Inner Maximization: A 1-Lipschitz neural network is trained to approximate the Wasserstein distance between the purified untrusted data and the reference data.
  2. Outer Minimization: Perturbations are added to the untrusted data to minimize this approximated Wasserstein distance, while also penalizing large perturbations to maintain data utility.

This min-max problem is solved iteratively using a two-step approach:

  • Step I: Update the 1-Lipschitz neural network (the "critic" in Wasserstein GAN terms) using the current perturbations.
  • Step II: Update the perturbations using gradient descent to minimize the objective function, guided by the current critic.

The framework is designed to be general, not relying on specific manipulation algorithms or desired properties.

Datasets

The paper evaluates Mendata using standard image datasets:

  • CIFAR-10: A widely used dataset for image classification.
  • TinyImageNet: A larger and more diverse dataset, used to test robustness across a wider range of classes and image complexities.
  • CIFAR-100: Used for data tracing experiments.

The authors also utilize a simulated detector to control the quality (precision and recall) of the reference dataset, allowing for a systematic evaluation of Mendata's performance under varying conditions of data cleanliness.

Key Insights and Ideas

  • Wasserstein Distance as a Distribution Alignment Metric: The paper highlights the effectiveness of Wasserstein distance for measuring and aligning feature distributions, which is crucial for removing hidden properties without altering the data's core utility.
  • Min-Max Optimization for Purification: Formulating purification as a min-max problem allows for a principled way to balance the competing goals of removing hidden properties (maximizing the distance to the manipulated distribution) and preserving data utility (minimizing perturbations).
  • Two-Step Iterative Solution: The proposed iterative approach effectively solves the complex min-max optimization problem, making the framework practical.
  • Generality of the Framework: Mendata's design is agnostic to the specific data manipulation technique, making it broadly applicable to various poisoning and tracing scenarios.
  • Reference Data Quality Matters: The effectiveness of Mendata is shown to be correlated with the precision of the detector used to identify the clean reference dataset. Higher precision leads to better purification.

Key Results

Mendata demonstrated strong performance in defending against both data poisoning and data tracing attacks:

  • Data Poisoning: Mendata significantly reduced the Attack Success Rate (ASR) for both targeted and backdoor attacks on CIFAR-10 and TinyImageNet, often achieving comparable accuracy to baseline methods while drastically lowering the ASR. For instance, in some scenarios, Mendata achieved near-zero ASR with minimal impact on accuracy.
  • Data Tracing: Mendata successfully defeated the Radioactive Data tracing technique on CIFAR-100 and TinyImageNet. It reduced the detection probability (p < 0.05) to near zero, indicating that the data contributor could no longer reliably detect the use of their marked data.
  • Comparison to Baselines: Mendata consistently outperformed or matched existing purification methods like Random Noise and Friendly Noise, and was effective even when the reference dataset contained a small fraction of poisoned data. It also showed robustness against adaptive manipulation strategies.

Limitations and Future Work

  • Detector Dependency: Mendata's performance is contingent on the quality (specifically, precision) of the detector used to identify the clean reference dataset. While it works well even with imperfect detectors, a very low-precision detector would limit its effectiveness.
  • Image Data Focus: The current framework is primarily demonstrated on image data.
  • Computational Cost: The iterative optimization process can be computationally intensive.

The authors suggest future work could involve:

  • Extending Mendata to other data modalities beyond images.
  • Developing purification methods with formal guarantees of effectiveness.
  • Further exploring adaptive training-data manipulation strategies and Mendata's robustness against them.

Summary updated

Abstract

Untrusted data used to train a model might have been manipulated to endow the learned model with hidden properties that the data contributor might later exploit. Data purification aims to remove such manipulations prior to training the model. We propose Mendata, a novel framework to purify manipulated training data. Starting from a small reference dataset in which a large majority of the inputs are clean, Mendata perturbs the training inputs so that they retain their utility but are distributed similarly (as measured by Wasserstein distance) to the reference data, thereby eliminating hidden properties from the learned model. A key challenge is how to find such perturbations, which we address by formulating a min-max optimization problem and developing a two-step method to iteratively solve it. We demonstrate the effectiveness of Mendata by applying it to defeat state-of-the-art data poisoning and data tracing techniques.
BibTeX
@misc{huang2023mendata,
  author = {Zonghao Huang and Neil Gong and Michael K. Reiter},
  title = {{Mendata: A Framework to Purify Manipulated Training Data}},
  year = {2023},
  eprint = {2312.01281},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.01281},
}
Figure