Federated Learning (FL) has emerged as a cornerstone for privacy-preserving distributed machine learning, allowing multiple clients to collaboratively train a model without sharing their raw data. However, the transmission of model updates still poses significant privacy risks, as sophisticated attackers can potentially reconstruct sensitive local information from these gradients. Differential Privacy (DP) is the standard defense, injecting noise into updates to provide mathematical privacy guarantees. Yet, traditional DP-FL faces a steep trade-off: excessive noise degrades model utility, while insufficient noise compromises privacy.
The Challenge of Constant Perturbation
Standard DP-FL implementations typically apply a constant level of Gaussian noise across all training rounds. This "one-size-fits-all" approach is often sub-optimal. In early training stages, high noise can lead to divergence, preventing the model from establishing a clear learning trajectory. Conversely, as training progresses and the cumulative privacy budget is consumed, maintaining high noise levels can cause the model to oscillate or diverge from the optimal solution.
As illustrated in the system model above, the server aggregates local updates ($\tilde{\omega}_k$) to produce a global model ($\omega$). The goal is to protect these updates from an attacker while ensuring the final global model remains accurate.
Proposed Methodology: Time-Varying DP Perturbation
Researchers have proposed a novel Time-Varying DP Perturbation Mechanism to address these inefficiencies. Instead of fixed noise, the noise variance $\Theta(m)$ for the $m$-th round follows a geometric series:
$$\Theta(m) = \vartheta^{m-1}\sigma^2$$
Where $\sigma^2$ is the initial noise variance and $\vartheta$ is a scaling factor. By adjusting $\vartheta$, the system can either increase ($\vartheta > 1$) or decrease ($\vartheta < 1$) the noise amplitude over time. This flexibility allows the mechanism to be tailored to specific model architectures; for instance, empirical results suggest that decreasing noise ($\vartheta < 1$) often benefits Convolutional Neural Networks (CNNs), while increasing noise ($\vartheta > 1$) can be more effective for Support Vector Machines (SVMs).
Experimental Results and the "U-Shaped" Loss Curve
The mechanism was evaluated across several benchmarks, including MNIST, CIFAR-10, and Fashion-MNIST. A critical finding is the existence of a "U-shaped" loss curve in DP-FL. Unlike standard machine learning where more training generally leads to better results, DP-FL reaches a point of diminishing returns where cumulative noise begins to outweigh the benefits of further aggregation.
As shown in the experimental data, there is an optimal number of aggregation rounds ($M^*$) that minimizes the loss and maximizes accuracy. The proposed time-varying mechanism consistently outperforms standard constant-noise Gaussian and Laplacian mechanisms. Specifically, by starting with smaller noise levels, the model can converge faster and reach higher peak accuracy before the privacy budget is exhausted.
Conclusion
By strategically varying noise amplitude, this new approach provides a more nuanced balance between privacy and utility. It not only achieves superior accuracy and faster convergence but also demonstrates robust defense against Membership Inference Attacks (MIA), reducing attacker success rates from over 99% to under 60%. Future work will focus on automating the selection of the scaling factor $\vartheta$ and extending theoretical convergence proofs to more complex architectures like Transformers.
Summary updated