Déjà Vu: an empirical evaluation of the memorization properties of ConvNets

Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Hervé JégouarXiv 2018 · OpenReview

Abstract

Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate overfitting. This paper considers the related question of "membership inference", where the goal is to determine if an image was used during training. We consider it under three complementary angles. We show how to detect which dataset was used to train a model, and in particular whether some validation images were used at train time. We then analyze explicit memorization and extend classical random label experiments to the problem of learning a model that predicts if an image belongs to an arbitrary set. Finally, we propose a new approach to infer membership when a few of the top layers are not available or have been fine-tuned, and show that lower layers still carry information about the training samples. To support our findings, we conduct large-scale experiments on Imagenet and subsets of YFCC-100M with modern architectures such as VGG and Resnet.

Introduction

Deep learning models are notorious for their ability to memorize training data. This raises significant privacy and intellectual property concerns: if a model memorizes its training data, an attacker might be able to query the model to determine if a specific sensitive image was used to train it—a "membership inference" attack. While previous work explored this on small datasets like CIFAR or required access to the model's final output layer, this study addresses three practical questions for modern, large-scale vision models:

  1. Dataset Inference & Leakage: Can we detect if a specific dataset (like a public validation set) was secretly used to train a model to inflate benchmark scores?
  2. Explicit Capacity: What is the hard upper bound of a modern ConvNet's ability to explicitly memorize a massive list of random images?
  3. Partial-Layer Attacks: If a model is pre-trained on sensitive data and then fine-tuned on a new task, do the remaining intermediate layers still leak membership information?

Methodology

The authors tackle these questions using three distinct approaches:

  • Explicit Memorization: Standard ConvNets (like ResNet and VGG) are repurposed as binary classifiers. The network is trained to output 1 for a specific set of $n$ "in" images and 0 for a massive pool of unseen "out" images.
  • Dataset Detection & Leakage: The Kolmogorov-Smirnov (K-S) test is used to compare the cumulative distribution of a model's confidence scores. Because models are generally more confident on data they've seen, the K-S distance can statistically prove if a batch of images belongs to the training distribution or an unseen validation distribution.
  • Partial-Layers Membership Inference: To attack a fine-tuned model where the top layers are missing, the attacker takes the available lower layers, retrains the missing top layers using public data, and then applies standard membership inference heuristics to guess if an image was in the original training set.

Datasets

To ensure rigorous testing without accidental overlap, datasets were heavily sanitized using GIST descriptors to remove near-duplicates. The primary datasets used include ImageNet subsets (Imnet1k and Imnet22k), YFCC-100M (used to sample negative images for large-scale experiments), and Tiny Images for small-scale capacity experiments on custom "TinyNet" architectures.

Key Insights

A primary finding is that data augmentation is a powerful privacy shield. Techniques like random cropping and flipping drastically reduce a network's ability to explicitly memorize images. By forcing the network to recognize augmented variations, it learns generalizable features rather than exact pixel patterns.

Memorization capacity of TinyNet architectures under various data augmentation strategies.

Furthermore, the study shows that cheating on benchmarks leaves a clear signature. The K-S test can detect validation leakage with high confidence using only a handful of leaked images. Crucially, memory lives in the lower layers, too; even after fine-tuning on an unrelated task, intermediate layers retain a "signature" of the original training data. Finally, ConvNets exhibit smooth capacity degradation rather than a sharp drop-off as the number of target images increases.

Key Results

  • Leakage Detection: If a model is trained on the standard training set plus just 10 images per class from the validation set, the K-S test detects the leakage with extreme statistical significance ($p < 10^{-4}$).
  • Explicit Memorization: A ResNet-18 can explicitly memorize 100,000 images, but data augmentation (flips and crops) doubles the required training time and prevents perfect memorization.
  • Membership Inference (Full Model): On a ResNet-101 trained without data augmentation, attackers can guess membership with 90.4% accuracy. With flip/crop augmentation, this drops to 68.0%.
Training dynamics of ResNet-101 for explicit memorization tasks with and without augmentation.
  • Membership Inference (Partial Layers): Attacking just the 4th stage of a ResNet-101 (without the final classification block) yields an accuracy of 53.1%. While lower, this is still strictly better than random guessing (50%), proving that intermediate layers leak membership information.

Limitations and Future Work

The partial-layers attack suffers a significant accuracy drop compared to full-network attacks. Interestingly, "shadow models" underperformed in the partial-layers setting because aligning intermediate activations across different networks is highly complex. Future work will further explore the intersection of standard regularization techniques and formal privacy guarantees, as data augmentation acts as a highly effective, practical defense against membership inference.

Summary updated

Abstract

Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate overfitting. This paper considers the related question of "membership inference", where the goal is to determine if an image was used during training. We consider it under three complementary angles. We show how to detect which dataset was used to train a model, and in particular whether some validation images were used at train time. We then analyze explicit memorization and extend classical random label experiments to the problem of learning a model that predicts if an image belongs to an arbitrary set. Finally, we propose a new approach to infer membership when a few of the top layers are not available or have been fine-tuned, and show that lower layers still carry information about the training samples. To support our findings, we conduct large-scale experiments on Imagenet and subsets of YFCC-100M with modern architectures such as VGG and Resnet.
BibTeX
@misc{sablayrolles2018deja,
  author = {Alexandre Sablayrolles and Matthijs Douze and Cordelia Schmid and Hervé Jégou},
  title = {{Déjà Vu: an empirical evaluation of the memorization properties of ConvNets}},
  year = {2018},
  eprint = {1809.06396},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.06396},
}
Figure