Abstract
We consider the problem of developing privacy-preserving machine learning algorithms in a distributed multiparty setting. Here, different parties own different parts of a dataset, and the goal is to learn a single classifier from the union of these datasets without compromising individual privacy. We propose a novel Private Stochastic Gradient Descent (PSGD) algorithm that optimizes the global objective directly using a two-layer noise mechanism. Unlike traditional local aggregation methods, our approach provides generalization guarantees that are independent of the number of parties and robust to data skew, ensuring high utility even in highly distributed or uneven environments.
Introduction
In the modern data landscape, valuable information is often siloed across institutions like banks and hospitals. While collaborative machine learning could unlock significant insights, privacy regulations and competitive interests prevent direct data pooling. This "multiparty classification problem" requires learning a global model without any party—or a central coordinator—accessing individual data points.
Previous solutions, such as those by Pathak et al. (2010), relied on aggregating locally trained models. However, these methods suffer from a major "Achilles' heel": their accuracy degrades significantly as the number of parties ($K$) increases or when data is unevenly distributed (skewed). We introduce PSGD to overcome these scaling and distribution challenges.
Methodology
PSGD shifts the paradigm from aggregating models to collaborative optimization of the global objective function.
Two-Layer Noise Strategy
To ensure differential privacy against a potentially curious central coordinator, each party adds two types of noise to their gradient updates:
- Fixed Noise ($\eta^k$): Sampled once per party to perturb the objective function.
- Iterative Noise ($\rho^k_t$): Sampled at every step of the gradient descent to prevent the coordinator from reverse-engineering private data through gradient differences.
Secure Aggregation
The noisy gradients are summed using an additively homomorphic encryption scheme. This ensures the central coordinator only observes the aggregate sum, never individual contributions, maintaining strict privacy boundaries.
Experimental Evaluation
We evaluated PSGD against Local Aggregation (LA) and a baseline Global Objective Perturbation (GOP) using simulated data and the Wisconsin Breast Cancer dataset.
Robustness to Scale and Skew
A key theoretical finding is that PSGD's excess error is independent of the number of parties ($K$). Empirically, as $K$ increased from 5 to 15 in simulations, LA's error rate climbed steadily, while PSGD remained stable.
Furthermore, PSGD handles data skew effectively. In scenarios where data ownership is highly uneven, LA's performance degrades sharply, whereas PSGD maintains consistent accuracy. This robustness is further validated on real-world medical data.
Conclusion
By optimizing the global objective directly, PSGD avoids the error accumulation inherent in averaging local models. It provides a scalable, robust solution for multiparty learning with $(\epsilon, \delta)$-differential privacy. Future work will explore extending this framework to non-linear models and deep neural networks, as well as reducing the cryptographic overhead of per-iteration secure summation.
Summary updated