Abstract
Training deep neural networks with Differential Privacy (DP) typically relies on noisy Stochastic Gradient Descent (DP-SGD), which often suffers from slow convergence and inefficient privacy budget utilization. This paper introduces DP-MAC, a novel approach that utilizes the Method of Auxiliary Coordinates (MAC) to decouple network layers. This decoupling enables the analytic calculation of sensitivity via Taylor expansions, allowing for direct perturbation of the objective function. While DP-MAC is highly competitive for shallow networks—achieving 0.95 accuracy on MNIST in 30 epochs compared to 120 for DP-SGD—the method currently faces challenges with vanishing gradients when scaling to deeper architectures.
Introduction
Deep learning models are prone to memorizing sensitive training data. Differential Privacy (DP) addresses this by injecting calibrated noise into the training process, with the noise level determined by the function's sensitivity. For complex, nested neural networks, calculating exact sensitivity is mathematically intractable, leading to the widespread adoption of DP-SGD. However, DP-SGD's reliance on many training steps results in cumulative privacy loss. DP-MAC seeks a fundamentally different training paradigm where sensitivity can be calculated more directly, potentially offering faster convergence and improved privacy-utility trade-offs.
Methodology
The core of DP-MAC is the adaptation of the Method of Auxiliary Coordinates (MAC) for private training.
- Layer Decoupling: MAC introduces auxiliary variables for activations at each layer, breaking the nested network into independent per-layer objective functions. Training alternates between updating auxiliary variables and network weights.
- Taylor Expansions and Perturbation: The decoupled objective functions are approximated using low-order Taylor expansions. The authors analytically calculate the sensitivity of the resulting polynomial coefficients and inject Gaussian noise directly into these coefficients.
- Privacy Accounting: The weight updates' cumulative privacy loss is tracked using the Moments Accountant method, while auxiliary variable updates do not require noise injection.
Experimental Results
The performance of DP-MAC was evaluated on MNIST (classification) and USPS (autoencoder reconstruction).
Shallow Networks
On the MNIST classification task using a single hidden layer, DP-MAC demonstrated significant efficiency. It achieved a test accuracy of 0.95 with a privacy budget of $\epsilon = 2$ in only 30 epochs. In contrast, standard DP-SGD required 120 epochs to reach the same accuracy level.
Deep Networks and Autoencoders
For deeper architectures, such as a 6-layer fully connected autoencoder on the USPS dataset, DP-MAC's performance was less robust compared to DP-SGD.
As shown in the figures above, DP-MAC (Figure 1) underperformed DP-SGD (Figure 2) across various privacy budgets. For instance, at $\epsilon = 2$, DP-MAC yielded a Mean Squared Error (MSE) of approximately 10.9 after 50 epochs, while DP-SGD achieved an MSE of 9.6 within 100 epochs.
Discussion and Limitations
The primary obstacle for DP-MAC in deeper networks is an inherent issue within the MAC algorithm: vanishing gradients. Observations showed that gradient updates in the first and last layers differed by up to four orders of magnitude. Additionally, while analytic sensitivity bounds were derived, they proved too loose for practical application, necessitating empirical coefficient clipping.
Despite these scaling challenges, DP-MAC provides a novel framework for computing sensitivities in neural networks. Future research may focus on stabilizing MAC for deep architectures or refining analytic bounds to further optimize the privacy-utility frontier.
Summary updated