Introduction
Deep Neural Networks (DNNs) are known to memorize training data, posing significant privacy risks. While previous research has focused on the memorization of entire samples or specific labels, this paper investigates a more granular phenomenon: unique feature memorisation. This occurs when a model learns a specific, non-generalizable feature that appears in only a single training instance.
Consider a medical imaging scenario where a patient's name or ID is accidentally left on an X-ray. If a model memorizes this text as a "shortcut" to a diagnosis, it creates a "feature leakage" risk. An adversary could query the model to determine if a specific individual was part of the training set, even without access to the original data.
Measuring Memorization
The authors propose a novel metric, the M score, to detect this leakage in black-box settings. Unlike previous methods, the M score does not require access to training data, model weights, or the labels of images containing the feature.
- White-box Score ($M_w$): Used when training data and labels are accessible. It compares model confidence on a "canary" (an image with the unique feature) against an image with random noise.
- Black-box Score ($M$): The primary contribution. It uses Out-of-Distribution (OOD) data to probe the model. By measuring the KL divergence between the model's outputs for OOD images with the unique feature versus those with random noise, the score quantifies the model's sensitivity to that specific feature.
Experimental Setup
The study utilizes three standard benchmark datasets: MNIST, Fashion-MNIST (F-MNIST), and CIFAR-10. To simulate feature leakage, the researchers "injected" a tiny 5x5 pixel letter 'A' into a single training image to act as a "canary."
The Regularization Paradox
A counterintuitive finding of this research is that standard techniques designed to prevent overfitting—such as Dropout, Batch Normalization, and Data Augmentation—do not necessarily prevent unique feature memorization. In fact, data augmentation often increases the likelihood of memorization. The authors hypothesize that augmentation creates multiple perturbed versions of the canary image, effectively providing the model with more opportunities to learn the spurious correlation between the unique feature and the label.
Furthermore, "early stopping" proves ineffective as memorization begins very early in the training process. In Multi-Layer Perceptrons (MLPs), the model was found to dedicate specific "neurons" to the unique feature, which remain dormant when presented with random noise.
Key Results
The study reveals widespread memorization across various architectures:
- Architecture Sensitivity: CNNs generally exhibit higher memorization scores than MLPs. On MNIST, a simple CNN had an average M score nearly ten times higher than an MLP (0.012 vs 0.0013).
- Batch Normalization Impact: On CIFAR-10, adding Batch Normalization to a CNN caused the memorization score to jump from 0.018 to 1.7.
- Influence Spikes: Injecting a unique feature into a "low-influence" sample (one the model typically learns easily) can turn it into a "highly influential" sample, causing its influence score to increase by several orders of magnitude.
Conclusion
This work highlights a significant and often overlooked privacy risk in machine learning. The discovery that common regularization techniques can exacerbate feature leakage suggests that current privacy-preserving methods may be insufficient. Future research is needed to develop training protocols that maintain high accuracy while actively mitigating the unintended memorization of unique features.
Summary updated