Abstract Differentially private learning tackles tasks where the data are private and the learning process is subject to differential privacy requirements. In real applications, however, some public data are generally available in addition to private data, and it is interesting to consider how to exploit them. In this paper, we study a common situation where a small amount of public data can be used when solving the Empirical Risk Minimization problem over a private database. Specifically, we propose Private-Public Stochastic Gradient Descent, which utilizes such public information to adjust parameters in differentially private stochastic gradient descent and fine-tunes the final result with model reuse. Our method keeps differential privacy for the private database, and empirical study validates its superiority compared with existing approaches.
Introduction
Training machine learning models with Differential Privacy (DP) often results in a significant "utility gap"—the model becomes much less accurate because of the noise added to protect individual data points. In Empirical Risk Minimization (ERM), DP algorithms must add noise to gradients, which scales poorly with high dimensionality and strict privacy requirements. However, in many real-world scenarios, researchers have access to a massive private database and a very small public one. This paper introduces PPSGD (Private-Public Stochastic Gradient Descent), a two-stage algorithm that leverages a small amount of non-sensitive "public" data to dynamically tune privacy parameters and fine-tune the final model, significantly improving accuracy without compromising the privacy of the sensitive dataset.
The PPSGD Framework
The proposed PPSGD framework operates in two distinct stages to maximize the utility of both private and public data sources.
- Stage 1: Differentially Private SGD with Dynamic Tuning. The model trains on the private database using Stochastic Gradient Descent. Crucially, the public data is used at each step to estimate the "true" gradient norm. If the public data suggests the noise is drowning out the signal, the algorithm dynamically increases the privacy budget ($\rho$) or adjusts the gradient clipping threshold ($C$). Because the public data is not sensitive, these adjustments do not leak private information.
- Stage 2: Model Reuse (Fine-tuning). After the private training is complete, the resulting model is treated as a "pre-trained" starting point. The algorithm then performs a final round of optimization using only the public data. To prevent overfitting to the tiny public set, biased regularization is employed, penalizing the model if it drifts too far from the privately-trained parameters.
The algorithm is built on the framework of Truncated Concentrated Differential Privacy (tCDP), which allows for more efficient tracking of the privacy "spent" across many iterations compared to standard $(\epsilon, \delta)$-DP.
Experimental Evaluation
The authors evaluated PPSGD on both synthetic and real-world benchmarks, including six classification datasets (e.g., adult-a, ipums-br, skin) and two regression datasets (cadata, stability). In these experiments, the "public" data was extremely small—often just 0.1% or 0.01% of the total training set.
As shown in the results table, PPSGD consistently outperformed state-of-the-art DP baselines (like Agd and SgdAdv) across almost all datasets. The performance gap was particularly noticeable in high-dimensional tasks and regression, where standard DP noise usually destroys model utility. For instance, on the cadata regression task with a privacy budget of $\epsilon=0.3$, PPSGD achieved a Mean Square Error of 0.1027, while the next best private baseline (AgdAvg) was significantly worse at 0.1253.
Key Insights
- Public Data as a Compass: The most innovative idea is using public data not as a primary training source, but as a "clean" reference to monitor the noise-to-signal ratio of the private training.
- Dynamic Clipping: PPSGD uses public data to shrink the clipping threshold as the model converges, reducing the amount of noise added in later stages.
- Efficiency: The method proved that you don't need a large public dataset to see benefits; even 50 public samples were enough to guide the training of 9,950 private samples effectively.
Conclusion
PPSGD demonstrates that even a tiny sliver of public data can be a powerful tool for bridging the utility gap in differentially private learning. By using public data for dynamic parameter tuning and final model refinement, PPSGD achieves superior accuracy while maintaining rigorous privacy guarantees. Future work may explore the impact of distribution shifts between public and private data and the application of these techniques to Local Differential Privacy.
Summary updated