Surrogate Model Extension (SME): A Fast and Accurate Weight Update Attack on Federated Learning

Junyi Zhu, Ruicong Yao, Matthew B. Blaschko

Abstract

Federated Learning (FL) is widely regarded as a privacy-preserving framework because it avoids the transmission of raw data, instead sharing only model weight updates. However, this paper introduces the Surrogate Model Extension (SME), a novel attack that demonstrates how these updates can be exploited to reconstruct private user data. By identifying a surrogate model along the training path, SME achieves data reconstruction up to 100x faster and with significantly higher accuracy than existing state-of-the-art methods, effectively shattering the "privacy illusion" of local training iterations.

Introduction

The fundamental promise of Federated Learning is that privacy is maintained by keeping data local. In this paradigm, clients perform several rounds of local training and only share the resulting "weight updates"—the difference between the initial and final model states—with a central server. It was long assumed that performing multiple local steps would obfuscate the individual gradients, which act as mathematical fingerprints of specific data points, thereby protecting user privacy.

Recent research into "gradient inversion" has challenged this assumption, showing that it is possible to reverse-engineer training data from these updates. However, existing attacks face a significant trade-off: they either fail when the number of local training steps is high or require "simulation-heavy" computations that demand massive GPU memory and time to emulate every discrete step taken by the client. The Surrogate Model Extension (SME) bridges this gap by providing a mathematically principled and computationally efficient method for data reconstruction.

Methodology: The Surrogate Model Extension

The core innovation of SME is the shift from step-by-step simulation to geometric approximation. Instead of attempting to replicate the entire sequence of local training iterations (as seen in methods like DLFA), SME identifies a single "surrogate" model state that represents the aggregate update.

1. Linear Interpolation

The authors propose that an effective surrogate model $\tilde{w}$ can be found on the linear path connecting the model's starting state ($w_0$) and its final state ($w_T$). This is defined by the interpolation: $$\tilde{w} = \alpha w_0 + (1 - \alpha) w_T$$ where $\alpha$ is a scalar between 0 and 1. This surrogate state serves as a proxy for the various points the model passed through during local training.

2. Joint Optimization

Standard inversion attacks typically optimize "dummy data" to produce a gradient that matches the observed update. SME improves upon this by simultaneously optimizing both the dummy data and the value of $\alpha$. This joint optimization allows the attack to dynamically find the most representative point in the parameter space for the specific update being analyzed.

3. Theoretical Foundation

The method is grounded in the observation that in a 2D parameter space, there always exists a point on the line between $w_0$ and $w_T$ where the gradient is perfectly parallel to the total weight update. While neural networks operate in high-dimensional spaces, the authors demonstrate that these models exhibit a "low-rank property." This means that most updates occur within a low-dimensional subspace, allowing the 2D geometric logic of SME to remain highly effective even for complex architectures.

Experimental Evaluation

The researchers evaluated SME across several standard image classification benchmarks and diverse architectures to test its robustness:

  • Datasets: FEMNIST (handwritten characters) and CIFAR100 (natural images).
  • Architectures: The study covered Multi-Layer Perceptrons (MLP), LeNet, two-layer CNNs, ResNet8, and Vision Transformers (ViT).

Key Results

  • Computational Efficiency: SME demonstrated a massive speedup, running up to 100x faster than the previous state-of-the-art simulation method (DLFA).
  • Memory Footprint: While simulation-heavy attacks often require over 100GB of GPU memory for complex models—rendering them impossible on standard hardware—SME maintained a constant memory cost of less than 3GB, regardless of the number of local training steps.
  • Reconstruction Quality: SME consistently outperformed baselines in Peak Signal-to-Noise Ratio (PSNR). On the FEMNIST dataset, it improved reconstruction quality by +2.5 to +3.2 PSNR compared to standard gradient inversion techniques.
  • Versatility: Unlike previous methods that struggled to converge on modern architectures, SME proved effective against Vision Transformers (ViT) and ResNets.

Discussion and Limitations

SME highlights a critical vulnerability in FL: increasing the number of local epochs does not necessarily "hide" data. Instead, it merely shifts the representative gradient to a different point in the parameter space—a point that SME is specifically designed to locate.

However, the method does have limitations. The current theoretical framework assumes standard Stochastic Gradient Descent (SGD). Adaptive optimizers like Adam, which introduce non-linearities into the update path, may interfere with the simple linear interpolation model. Additionally, like most gradient inversion attacks, SME’s efficacy decreases as the local batch size increases, making it harder to isolate individual images from a large aggregate.

Conclusion

The Surrogate Model Extension (SME) represents a significant advancement in the study of Federated Learning vulnerabilities. By leveraging the geometric properties of model updates, it provides a fast, memory-efficient, and highly accurate means of data reconstruction. This work serves as a vital reminder that sharing model weights is not a total solution for privacy, and the community must look toward more robust defenses beyond simple local iteration.

Summary updated

BibTeX
@inproceedings{zhu2023surrogate,
  author = {Junyi Zhu and Ruicong Yao and Matthew B. Blaschko},
  title = {{Surrogate Model Extension (SME): A Fast and Accurate Weight Update Attack on Federated Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=83arurBQg-f},
}
Figure