Efficient Hyperparameter Optimization for Differentially Private Deep Learning

Aman Priyanshu, Rakshit Naidu, Fatemehsadat Mireshghallah, Mohammad Malekzadeh

Training deep learning models with Differential Privacy (DP) requires a delicate balancing act. Every training run "consumes" a portion of the privacy budget, making traditional hyperparameter tuning methods like grid search both computationally expensive and privacy-draining. This research introduces a framework to efficiently navigate the tradeoff between model utility and privacy leakage using adaptive optimization techniques.

The Challenge of Private Hyperparameter Tuning

In standard deep learning, practitioners often perform exhaustive searches to find optimal hyperparameters such as learning rate or batch size. However, in Differentially Private Stochastic Gradient Descent (DPSGD), each access to the private dataset is quantified by a privacy loss parameter, epsilon ($\epsilon$).

The Pareto frontier illustrates the tradeoff between model utility and privacy. Desirable tradeoffs lie on the frontier, while suboptimal configurations fall into the undesirable region.

A naive grid search quickly exhausts the privacy budget or results in models with weak privacy guarantees because it treats every hyperparameter combination with equal importance, regardless of its performance. The goal is to find the "Pareto Frontier"—the set of hyperparameter configurations that provide the best possible utility for a given level of privacy.

Adaptive Optimization Framework

The proposed framework shifts from exhaustive search to an adaptive approach aimed at maximizing a custom Reward Function. This function balances utility (minimizing validation loss) and privacy (minimizing $\epsilon$):

$$reward = \alpha^U \cdot e^{-val\_loss} + \alpha^P \cdot e^{-\epsilon}$$

By adjusting the weights ($\alpha^U, \alpha^P$), practitioners can prioritize accuracy or privacy based on their specific requirements. The authors evaluated four strategies:

  1. Grid Search (Baseline): Non-adaptive and exhaustive.
  2. Evolutionary Optimization: A genetic algorithm that evolves hyperparameter "genomes" toward higher rewards.
  3. Bayesian Optimization: A probabilistic approach that balances exploration of the search space with exploitation of known high-performing regions.
  4. Reinforcement Learning (RL): An agent-based approach that learns to estimate rewards across the search space.

Key Results and Efficiency Gains

The adaptive methods significantly outperformed the grid search baseline across standard benchmarks like MNIST and CIFAR-10.

Comparison of training time across different optimization methods. Adaptive methods like Evolutionary and Bayesian optimization offer massive speedups over traditional Grid Search.

On the complex CIFAR-10 dataset, Evolutionary Optimization achieved superior rewards in just 11 hours, compared to the 150 hours required for Grid Search—a nearly 14x speedup. Furthermore, adaptive methods are inherently more private in a practical sense; they reduce the "re-visit" count of data samples by focusing the search on promising hyperparameter regions rather than wasting the privacy budget on suboptimal configurations.

Conclusion and Future Work

Adaptive optimization transforms hyperparameter tuning for private deep learning from a resource-intensive bottleneck into an efficient process. While this study focused on the noise multiplier and learning rate, future work could expand this framework to include Neural Architecture Search (NAS) and more complex privacy budget allocation strategies between the tuning and final training phases.

Summary updated

Abstract

Tuning the hyperparameters in the differentially private stochastic gradient descent (DPSGD) is a fundamental challenge. Unlike the typical SGD, private datasets cannot be used many times for hyperparameter search in DPSGD; e.g., via a grid search. Therefore, there is an essential need for algorithms that, within a given search space, can find near-optimal hyperparameters for the best achievable privacy-utility tradeoffs efficiently. We formulate this problem into a general optimization framework for establishing a desirable privacy-utility tradeoff, and systematically study three cost-effective algorithms for being used in the proposed framework: evolutionary, Bayesian, and reinforcement learning. Our experiments, for hyperparameter tuning in DPSGD conducted on MNIST and CIFAR-10 datasets, show that these three algorithms significantly outperform the widely used grid search baseline. As this paper offers a first-of-a-kind framework for hyperparameter tuning in DPSGD, we discuss existing challenges and open directions for future studies. As we believe our work has implications to be utilized in the pipeline of private deep learning, we open-source our code at https://github.com/AmanPriyanshu/DP-HyperparamTuning.
BibTeX
@misc{priyanshu2021efficient,
  author = {Aman Priyanshu and Rakshit Naidu and Fatemehsadat Mireshghallah and Mohammad Malekzadeh},
  title = {{Efficient Hyperparameter Optimization for Differentially Private Deep Learning}},
  year = {2021},
  eprint = {2108.03888},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.03888},
}
Figure