Abstract
Differentially private federated learning (FL) requires bounding the sensitivity of client updates to ensure privacy. The standard approach, clipping, projects updates onto an $\ell_2$ ball of radius $C$. However, clipping introduces complex biases and its convergence properties on non-Lipschitz objectives—common in deep learning—are poorly understood. This paper proposes DP-NormFedAvg, a simpler alternative that replaces clipping with normalization, effectively using only the unit vector of client updates. We demonstrate that normalization achieves the same order-wise convergence rate as standard FedAvg on smooth quasar-convex functions. Furthermore, by leveraging a strong-growth condition on client losses, we show that sensitivity can be reduced by a factor of $\mathcal{O}(1/m)$, where $m$ is the minimum number of samples per client, significantly improving iteration complexity. Empirical results on Fashion-MNIST and CIFAR datasets confirm that normalization consistently outperforms clipping by maintaining a superior signal-to-noise ratio.
Introduction
Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models on decentralized data residing on devices like smartphones or medical sensors. To provide formal privacy guarantees, Differential Privacy (DP) is typically integrated into the FL pipeline, usually by adding Gaussian noise to aggregated client updates.
The magnitude of this noise is proportional to the "sensitivity" of the updates—the maximum change a single user's data can cause. To bound this sensitivity, practitioners traditionally use clipping, where updates exceeding a threshold $C$ are scaled down. While effective for privacy, clipping introduces a bias that is difficult to quantify, especially in the presence of local training steps and non-Lipschitz loss functions. This work explores normalization as a robust alternative, discarding magnitude information entirely to simplify the optimization landscape and improve performance.
Methodology: From Clipping to Normalization
In standard DP-FL, a client update $u$ is processed using a clipping function: $$g_{clip} = u \cdot \min\left(1, \frac{C}{\|u\|_2}\right)$$ If the update's norm is below $C$, it remains unchanged; if it exceeds $C$, it is projected onto the surface of the $\ell_2$ ball.
In contrast, the proposed DP-NormFedAvg algorithm applies normalization to every update: $$g_{norm} = C \cdot \frac{u}{\|u\|_2}$$ By forcing every update to have an identical norm $C$, the algorithm focuses exclusively on the direction of the gradient. This approach eliminates the need to assume the objective function is Lipschitz-continuous, a common but often unrealistic assumption in deep learning theory.
Theoretical Insights
Convergence on Non-Lipschitz Objectives
The authors provide the first formal convergence analysis for both clipping and normalization on smooth quasar-convex functions. They show that DP-NormFedAvg maintains the convergence rates of standard FedAvg (modulo the noise term). This is a significant step forward, as it provides a theoretical basis for DP-FL in settings where gradients may grow significantly, such as in the early stages of training deep neural networks.
The Signal-to-Noise Ratio (SNR) Advantage
A critical insight of this work is the behavior of the Signal-to-Noise Ratio (SNR) as the model approaches an optimum. In clipping-based DP-FL, as the model converges, the magnitude of the true gradient $\|u\|$ naturally decreases. Eventually, $\|u\|$ falls well below the threshold $C$. However, the noise added for privacy remains proportional to $C$. Consequently, the signal vanishes into the noise, causing the model to stall.
Normalization prevents this "vanishing signal" problem. Because the update is always scaled back to $C$, the signal strength relative to the noise remains constant throughout the training process, allowing for finer convergence near the optimum.
Local Steps and Heterogeneity
The paper also quantifies the impact of local iterations ($E$). While performing more local steps can help the model move away from a poor initialization faster, it also amplifies the error caused by data heterogeneity (non-IID data). The authors provide a framework to balance these factors under the normalization regime.
Experimental Results
The researchers evaluated DP-NormFedAvg against standard clipping across several benchmarks, including Fashion-MNIST and CIFAR-10/100 (using ResNet-18 features).
- Accuracy Improvements: Normalization consistently yielded higher test accuracy. For instance, on CIFAR-100 with a privacy budget of $\epsilon=5$, normalization improved accuracy by 2.8% over clipping. On Fashion-MNIST, the gain was 2.1%.
- Optimization Stability: In synthetic quadratic experiments, normalization trajectories reached closer to the true functional optimum, whereas clipping trajectories were often diverted or halted by the static noise floor.
- Robustness to Heterogeneity: Even in highly non-IID settings (where clients only hold a few classes of data), normalization proved more resilient than clipping, maintaining a higher SNR across training rounds.
Conclusion
By replacing clipping with normalization, DP-NormFedAvg offers a more theoretically grounded and empirically performant approach to private federated learning. The method's ability to maintain a stable SNR and its convergence guarantees on non-Lipschitz functions make it a compelling choice for practical DP-FL implementations. Future research may explore adaptive scaling factors ($C$) that evolve during training to further optimize the privacy-utility tradeoff.
Summary updated