Neural networks memorise personal information from one sample

John Hartley, Pedro P. Sanchez, Fasih Haider, Sotirios A. TsaftarisScientific 2023 · DOI

Abstract: Deep neural networks (DNNs) have achieved high accuracy in diagnosing multiple diseases/conditions at a large scale. However, a number of concerns have been raised about safeguarding data privacy and algorithmic bias of the neural network models. We demonstrate that unique features (UFs), such as names, IDs, or other patient information can be memorised (and eventually leaked) by neural networks even when it occurs on a single training data sample within the dataset. We explain this memorisation phenomenon by showing that it is more likely to occur when UFs are an instance of a rare concept. We propose methods to identify whether a given model does or does not memorise a given (known) feature. Importantly, our method does not require access to the training data and therefore can be deployed by an external entity. We conclude that memorisation does have implications on model robustness, but it can also pose a risk to the privacy of patients who consent to the use of their data for training models.

Summary

Neural networks can memorize unique, private features—such as a patient’s name on a medical scan—from a single training example. This "Unique Feature Memorisation" (UFM) occurs early in training, persists despite standard regularization, and can be detected by external auditors without access to the original training data.

Introduction: The Problem of Unique Feature Memorisation

While deep learning models are designed to learn generalizable patterns, they often memorize specific training data. Previous research has focused on the memorization of entire samples or labels. This study investigates Unique Feature Memorisation (UFM): the unintended learning of specific, rare attributes (e.g., a name, ID number, or watermark) that appear only once in a dataset.

UFM presents two primary risks:

  1. Privacy: Memorized personal information could be extracted from model weights by adversaries.
  2. Robustness: Models may learn "shortcuts." For instance, a model might associate the name "JOHN" on a scan with a specific diagnosis rather than identifying the actual medical pathology, leading to failures in clinical deployment.

Methodology

To quantify this phenomenon, the authors introduce the M score, a metric that measures a model's reliance on a specific unique feature ($z_u$). The M score calculates the difference in model confidence when the unique feature is present versus when it is absent.

The experimental pipeline: unique features (like the name 'John') are introduced into training samples, and the resulting DNN's attention is analyzed via GradCAM.

The study evaluates UFM across three auditing scenarios of increasing difficulty:

  • White-box: The auditor has access to training data and the specific label associated with the feature.
  • Grey-box: The auditor has training data but not the specific class association.
  • Black-box: The most realistic scenario; the auditor has no training data and must use a proxy dataset to test sensitivity.

Experimental Setup

Hypotheses were tested across several datasets, including synthetic benchmarks (Two Moons), standard vision tasks (F-MNIST, CIFAR-10, Celeb-A), and real-world medical data. For the CheXpert chest X-ray dataset, a unique name ("JOHN") was added to a single training image to simulate a failure in data anonymization.

Analysis of Memorisation

  • Rarity as a Driver: Memorization is most prevalent when a feature represents a "rare concept" (e.g., text appearing on only one image in a set of text-free images).
  • Early Onset: UFM occurs within the first few epochs, often before the model has generalized to the broader dataset.
  • Resistance to Regularization: Standard techniques like Dropout, Weight Decay, and Data Augmentation fail to prevent UFM.
  • Verbatim Capture: Models are sensitive to exact pixel-level representations; even minor alterations to the unique feature significantly reduce the M score, suggesting the information is captured verbatim.

Results and Visual Evidence

In the CheXpert dataset, 60% of models memorized the single instance of the name "JOHN." Crucially, the Black-box M score correlated strongly with White-box ground truth, suggesting that external regulatory bodies can successfully audit models for privacy leaks without access to proprietary training data.

GradCAM heatmaps illustrating the difference between a model that has memorized a unique feature (left, focusing on the name 'JOHN') and one that has generalized to medical pathology (right, focusing on the lungs).

Visual analysis using GradCAM heatmaps confirms that for memorized models, the network's attention is focused entirely on the unique feature rather than the relevant medical information, confirming the model has learned a shortcut rather than a diagnostic pattern.

Discussion and Future Work

A current limitation is that the M score requires prior knowledge of the suspected unique feature. Future research should focus on:

  • Automated "privacy filters" to sanitize data before inference.
  • Robust training pipelines that prevent unique features from being learned.
  • Investigating UFM in Large Language Models (LLMs), where unique strings may be memorized from single documents.

Summary updated

Abstract

Deep neural networks (DNNs) have achieved high accuracy in diagnosing multiple diseases/conditions at a large scale. However, a number of concerns have been raised about safeguarding data privacy and algorithmic bias of the neural network models. We demonstrate that unique features (UFs), such as names, IDs, or other patient information can be memorised (and eventually leaked) by neural networks even when it occurs on a single training data sample within the dataset. We explain this memorisation phenomenon by showing that it is more likely to occur when UFs are an instance of a rare concept. We propose methods to identify whether a given model does or does not memorise a given (known) feature. Importantly, our method does not require access to the training data and therefore can be deployed by an external entity. We conclude that memorisation does have implications on model robustness, but it can also pose a risk to the privacy of patients who consent to the use of their data for training models.
BibTeX
@inproceedings{hartley2023neural,
  author = {John Hartley and Pedro P. Sanchez and Fasih Haider and Sotirios A. Tsaftaris},
  title = {{Neural networks memorise personal information from one sample}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-48034-3},
}
Figure