See through Gradients: Image Batch Recovery via GradInversion

Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, Pavlo Molchanov

Abstract

Training deep neural networks requires gradient estimation from data batches to update parameters. Gradients per parameter are averaged over a set of data and this has been presumed to be safe for privacy-preserving training in joint, collaborative, and federated learning applications. Prior work only showed the possibility of recovering input data given gradients under very restrictive conditions—a single input point, or a network with no non-linearities, or a small 32x32 px input batch. In this work, we introduce GradInversion, using which input images from a larger batch (8 - 48 images) can also be recovered for large networks such as ResNets (50 layers), on complex datasets such as ImageNet (1000 classes, 224x224 px). We formulate an optimization task that converts random noise into natural images, matching gradients while regularizing image fidelity. We also propose an algorithm for target class label recovery given gradients and a group consistency regularization framework. We show that gradients encode a surprisingly large amount of information, such that all the individual images can be recovered with high fidelity via GradInversion.

Introduction

In collaborative and federated learning, multiple participants train a shared model by exchanging gradients rather than raw data. For a long time, the community assumed that averaging these gradients over a batch of data provided a "privacy buffer." While researchers had shown it was possible to recover a single image from a gradient, doing so for large batches of high-resolution images (like 224x224 px ImageNet samples) on deep architectures like ResNet-50 was considered computationally infeasible. GradInversion challenges this safety assumption, demonstrating exactly how much information is leaked when sharing averaged gradients.

The GradInversion Framework

The authors introduce GradInversion, an optimization-based framework that transforms random noise into a reconstruction of the original training batch. The method relies on three main pillars:

  1. Batch Label Restoration: Before reconstructing pixels, a "one-shot" algorithm identifies the ground-truth labels in the batch by analyzing the signs and magnitudes of the gradients in the final fully connected layer.
  2. Fidelity (Realism) Regularization: To ensure recovered images look like real objects, the method uses Batch Normalization (BN) statistics as a prior, steering the optimization toward natural-looking image manifolds.
  3. Group Consistency Regularization: Independent optimization runs often produce the same features in different positions due to CNN translational invariance. GradInversion runs multiple "agents" in parallel, uses image registration (RANSAC-flow) to align their outputs, and enforces a "consensus" image to regularize the group, significantly reducing artifacts.
The GradInversion pipeline, showing the image generator and group registration process.

Experimental Results

The primary evaluation is conducted on the ImageNet dataset using a ResNet-50 architecture. GradInversion successfully recovers batches of 8, 16, 32, and even 48 images, whereas previous state-of-the-art methods typically failed beyond a batch size of 1.

Overview of the GradInversion attack and examples of recovered images from averaged gradients.

Quantitatively, on ImageNet (batch size 8), GradInversion achieved an LPIPS perceptual similarity score of 0.484, significantly outperforming prior methods like DLG (1.319) and Inverting Gradients (0.749). The authors also introduced the Image Identifiability Precision (IIP) metric, showing that even at a batch size of 48, roughly 28% of images can still be correctly identified from the global dataset based on their reconstructions.

Discussion and Conclusion

Gradients are high-bandwidth proxies that encode dense information about input data. Even when averaged over 48 different images, the "signal" of individual images remains detectable. The "vulnerable population"—images that produce large gradients—is particularly susceptible to recovery. These findings suggest that current gradient-sharing protocols in federated learning are insufficient for privacy and necessitate more robust defense mechanisms, such as gradient clipping or sophisticated differential privacy techniques.

Summary updated

Abstract

Training deep neural networks requires gradient estimation from data batches to update parameters. Gradients per parameter are averaged over a set of data and this has been presumed to be safe for privacy-preserving training in joint, collaborative, and federated learning applications. Prior work only showed the possibility of recovering input data given gradients under very restrictive conditions - a single input point, or a network with no non-linearities, or a small 32x32 px input batch. Therefore, averaging gradients over larger batches was thought to be safe. In this work, we introduce GradInversion, using which input images from a larger batch (8 - 48 images) can also be recovered for large networks such as ResNets (50 layers), on complex datasets such as ImageNet (1000 classes, 224x224 px). We formulate an optimization task that converts random noise into natural images, matching gradients while regularizing image fidelity. We also propose an algorithm for target class label recovery given gradients. We further propose a group consistency regularization framework, where multiple agents starting from different random seeds work together to find an enhanced reconstruction of original data batch. We show that gradients encode a surprisingly large amount of information, such that all the individual images can be recovered with high fidelity via GradInversion, even for complex datasets, deep networks, and large batch sizes.
BibTeX
@inproceedings{yin2021see,
  author = {Hongxu Yin and Arun Mallya and Arash Vahdat and Jose M. Alvarez and Jan Kautz and Pavlo Molchanov},
  title = {{See through Gradients: Image Batch Recovery via GradInversion}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=xCPnCd70q7},
}
Figure