Abstract
Differential Privacy (DP) often introduces a significant "utility tax" due to the noise required to protect sensitive data. Traditional methods rely on hand-tuned noise schedules that are often sub-optimal and difficult to generalize. We present Learning-to-Protect (L2P), a meta-learning framework that automates the design of privacy protection strategies. L2P employs a model-based "protector" consisting of two Recurrent Neural Networks (RNNs) to adaptively allocate privacy budgets and denoise gradients. By meta-training on non-private auxiliary data, L2P learns to optimize the utility-privacy trade-off, significantly outperforming state-of-the-art baselines in high-privacy regimes.
Introduction
Training machine learning models with Differential Privacy involves a fundamental trade-off between model utility and data privacy. The privacy budget, $\epsilon$, acts as a finite resource that is depleted as noise is added to gradients during training. Existing approaches typically use fixed heuristics or simple adaptive rules for noise injection. However, these methods fail to account for the complex dynamics of the optimization landscape and often require manual tuning that might itself leak private information. This work introduces L2P, which treats the discovery of optimal privacy strategies as a learning problem, leveraging auxiliary data to pre-train a protector model.
Methodology
The L2P framework introduces a "Protector" module that replaces standard noise injection mechanisms.
The Model-Based Protector
The protector is composed of two specialized RNNs:
- The Scheduler ($\sigma$): This component monitors the training state and dynamically determines the noise level for each gradient update. It learns to strategically preserve the privacy budget for the most impactful training phases.
- The Projector ($\pi$): After noise is added, the projector RNN processes the noisy gradient to predict an optimal parameter update. It effectively learns to "denoise" the signal, extracting meaningful patterns even in high-noise environments.
Privacy Accounting and Meta-Optimization
To facilitate efficient training, the authors propose Constrained Truncated Concentrated DP (ctCDP), a metric that allows for linear accumulation of privacy costs. The protector is meta-trained using a Lagrangian method on public auxiliary datasets. The objective is to minimize the final model loss while strictly adhering to a predefined privacy budget constraint.
Experimental Evaluation
L2P was evaluated on diverse tasks, including logistic regression on IPUMS census data and Multi-Layer Perceptrons (MLP) on MNIST35. A key focus was generalization: training the protector on one dataset (e.g., Brazilian census) and testing it on another (e.g., US census).
Results
L2P consistently outperformed baselines such as Adaptive Gradient Descent (AGD) and Moments Accountant (SGD-MA), particularly in low-budget (high-privacy) scenarios.
Key findings include:
- Enhanced Utility: L2P achieves higher accuracy by learning to filter noise through its projector RNN.
- Adaptive Strategies: The learned schedules often prioritize noise reduction in later stages of training to ensure convergence.
- Robust Generalization: Protectors trained on auxiliary tasks successfully transfer their strategies to new, private datasets, demonstrating that privacy-preserving patterns are learnable across domains.
Conclusion
Learning-to-Protect (L2P) represents a shift from manual to automated privacy engineering. By framing privacy protection as a meta-learning task, L2P discovers adaptive strategies that significantly improve the utility of DP-trained models. While dependent on the availability of relevant auxiliary data, L2P provides a powerful tool for deploying high-utility models under strict privacy constraints. Future work will explore private meta-optimization to reduce reliance on public datasets.
Summary updated