SSGD: A safe and efficient method of gradient descent

Jinhuan Duan, Xianxian Li, Shiqi Gao, Jinyan Wang, Zili Zhong

Abstract

Gradient sharing in distributed machine learning environments, such as Federated Learning, introduces significant privacy risks where attackers can reconstruct raw training data from shared gradient updates. We present Super Stochastic Gradient Descent (SSGD), a novel optimization method designed to mitigate these risks. By normalizing neuron-level gradients into unit vectors, SSGD effectively masks the gradient magnitude, a critical component for data reconstruction attacks. Our evaluation demonstrates that SSGD provides robust security against state-of-the-art attacks while maintaining model accuracy, training stability, and computational efficiency comparable to standard optimizers.

Introduction

In distributed machine learning, multiple participants collaboratively train a model by sharing local gradient updates instead of raw data. While intended to preserve privacy, recent studies have shown that shared gradients are highly vulnerable to reconstruction attacks like Deep Leakage from Gradients (DLG). These attacks can reverse-engineer training images and labels with high precision.

Existing countermeasures often come with trade-offs: secure aggregation is often incompatible with standard CNNs, homomorphic encryption is computationally prohibitive, and Differential Privacy (DP) typically degrades model performance. SSGD aims to bridge this gap by providing a mathematically grounded defense that prevents data reconstruction without sacrificing the utility of the trained model.

Methods

The core of Super Stochastic Gradient Descent (SSGD) lies in two primary mechanisms:

  1. Unit Vector Conversion: SSGD transforms the raw gradient vector of each neuron or convolution kernel into a unit vector. By normalizing the gradient, the modulus length (magnitude) is discarded. This transformation breaks the bijective relationship between the gradient and the input data.
  2. Multi-Sample Aggregation: To mitigate the "super-randomness" introduced by unit vector normalization, which might otherwise destabilize the training process, SSGD aggregates unit gradients across multiple samples within a batch before performing parameter updates.

Mathematically, by removing the modulus length, the system of equations an attacker must solve becomes underdetermined. With $d$ equations and $d+1$ unknowns, exact reconstruction becomes impossible, providing a fundamental layer of privacy at the neuron level.

Results

Accuracy and Convergence

Empirical evaluations on the MNIST dataset show that SSGD and its momentum-based variant (SSGDm) achieve performance parity with standard optimizers like SGD and Adam. As shown in Figure 2, SSGDm converges rapidly to approximately 99% test accuracy, demonstrating that the loss of gradient magnitude information does not hinder the model's ability to learn effectively.

Test accuracy of SSGD compared to standard optimizers on MNIST.

Robustness and Scalability

SSGD exhibits superior scalability, maintaining stability even at very large batch sizes (e.g., 8192) where standard methods often require complex learning rate tuning. Furthermore, in high-noise environments simulating Differential Privacy ($\epsilon=0.2$), SSGDm maintained 84.81% accuracy, significantly outperforming Adam (80.74%) and standard SGDm (37.45%).

Defense Against Reconstruction Attacks

The primary strength of SSGD is its resilience against gradient-matching attacks. While DLG and iDLG can easily recover recognizable images from standard gradients, they fail completely against SSGD. As illustrated in Figure 6, attempts to reconstruct an MNIST digit from SSGD gradients result in persistent static noise across hundreds of iterations.

Reconstruction attack (DLG) failing to recover an image from SSGD gradients.

Conclusion

SSGD provides a lightweight and effective solution for privacy-preserving distributed learning. By focusing on the vulnerability inherent in gradient magnitudes and employing unit vector normalization, it thwarts sophisticated reconstruction attacks. While current evaluations focus on classic architectures like LeNet-5, future work will explore the application of SSGD to modern, deep architectures such as Transformers and ResNets to further validate its generalizability and stability.

Summary updated

BibTeX
@article{duan2021ssgd,
  author = {Jinhuan Duan and Xianxian Li and Shiqi Gao and Jinyan Wang and Zili Zhong},
  title = {{SSGD: A safe and efficient method of gradient descent}},
  journal = {Secur. Commun. Networks},
  year = {2021},
  doi = {10.1155/2021/5404061},
}
Figure