Abstract
Presented at AAAI-24, this research explores a critical vulnerability in distributed learning: the ability to reconstruct private training data from shared gradients. While distributed learning frameworks like Federated Learning are often touted for their privacy-preserving properties, the authors demonstrate a "High-Fidelity Gradient Inversion" attack that can recover high-resolution images even in challenging, large-batch scenarios. By combining an evolutionary algorithm for label inference with a stepwise optimization process, the proposed method significantly outperforms existing state-of-the-art attacks.
Introduction: The Privacy Gap in Distributed Learning
In distributed learning, clients train models locally and share only the resulting gradients with a central server. This mechanism is intended to keep raw data private. However, Gradient Inversion Attacks (GIAs) attempt to reverse-engineer these gradients to recreate the original training images. Existing GIAs face significant hurdles, including failure on high-resolution images, performance degradation as batch sizes increase, and difficulty handling duplicate labels within a batch. This paper addresses these "pain points" to prove that sharing gradients is not a guarantee of data privacy.
Methodology
The authors introduce a two-stage framework designed to overcome the limitations of previous reconstruction methods.
Stage I: Evolutionary Label Inference
To reconstruct data, an attacker must first identify the labels in the training batch. Standard methods struggle when multiple images share the same label. The authors innovate by using a small set of auxiliary data (publicly available images) to model the Coefficient of Variation (CV) of features. They then employ an evolutionary algorithm that uses this CV information to "evolve" a label distribution that precisely matches the shared gradients, enabling accurate recovery even with duplicate labels.
Stage II: Stepwise Gradient Inversion Attack
Once labels are known, the image reconstruction begins through a refined optimization process:
- Stepwise Optimization: Instead of matching all gradients at once, the attack introduces gradients layer-by-layer, starting from the deepest layers (rich in semantic content) and moving toward the shallow layers (rich in texture).
- Gradient Dropout: To solve the "spatial offset" problem where recovered images appear shifted or ghostly, the authors use random dropout on the gradients during optimization. This is significantly more computationally efficient than previous image alignment techniques like RANSAC-flow.
- Loss Scheduling: A phased approach for image priors (Total Variation and Batch Normalization statistics) is used to focus first on content recovery and then on polishing image realism.
Key Results and Evaluation
The proposed method, Step-Wise GradInv, sets new benchmarks for reconstruction quality on the ImageNet dataset using ResNet-18 and ResNet-34 architectures.
As illustrated in the figure, Step-Wise GradInv produces reconstructions with significantly higher fidelity than previous methods. Quantitatively, on ImageNet with a batch size of 8, it achieved an LPIPS score of 0.33 (outperforming GradInversion's 0.39) and a PSNR of 15.12 (compared to 13.22 for the previous state-of-the-art). Notably, the attack remains effective even at a batch size of 64, a scenario where most previous methods produce only unrecognizable noise. Furthermore, the attack proved robust against common defenses such as 90% gradient pruning and additive Laplacian noise.
Conclusion and Future Work
This work highlights a severe privacy risk in distributed learning. By demonstrating that high-resolution images can be recovered even from large batches, the authors underscore the need for more robust privacy protections. They suggest that because deep layers leak the most semantic information, future defense research should focus on specifically protecting or "bottlenecking" the gradients of deeper layers to preserve privacy without compromising model utility.
Summary updated