DP-NormFedAvg: Normalizing Client Updates for Privacy-Preserving Federated Learning

Rudrajit Das, Abolfazl Hashemi, Sujay Sanghavi, Inderjit S. Dhillon

Abstract

There is a significant gap in the theoretical understanding of differentially private federated learning (FL) when dealing with non-Lipschitz objective functions—settings where gradient norms are not naturally bounded. Standard approaches rely on a "clipping" operation to bound the sensitivity of client updates, but this introduces a complex bias that depends on the clipping threshold and the number of local steps. For non-Lipschitz functions, choosing an optimal threshold is non-trivial and often requires extensive tuning. In this paper, we provide the first convergence results for private FL on smooth convex objectives for a general clipping threshold without assuming Lipschitzness. Furthermore, we propose DP-NormFedAvg, a simpler alternative that replaces clipping with normalization. By discarding magnitude information and using only scaled unit vectors, we theoretically and empirically demonstrate that normalization achieves superior convergence and a better signal-to-noise ratio (SNR) than traditional clipping-based methods.

Introduction

Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models on decentralized data residing on client devices, such as mobile phones or IoT sensors. To protect the privacy of individual participants, Differential Privacy (DP) is often integrated into the FL workflow. The standard mechanism, DP-FedAvg, involves clients "clipping" their local model updates to a fixed $\ell_2$ radius $C$ before adding Gaussian noise to the aggregate.

However, this approach faces two significant hurdles. First, most existing convergence proofs rely on the "Lipschitz assumption," which assumes gradients never exceed a certain bound. In practice, many common loss functions are not Lipschitz, making the bias introduced by clipping difficult to analyze. Second, clipping creates a fundamental tension: a low threshold $C$ introduces high bias, while a high threshold requires adding massive noise to maintain privacy. This paper introduces DP-NormFedAvg to resolve these issues by shifting the focus from clipping to normalization.

The Limitations of Clipping

In traditional DP-FL, the clipping operation only scales down updates that exceed the threshold $C$. If an update is already smaller than $C$, it remains unchanged. While this preserves some magnitude information, it creates a vanishing signal-to-noise ratio (SNR) as the model approaches the optimum. As training progresses, the norm of the updates naturally shrinks; however, the noise added for privacy remains proportional to $C$. Eventually, the signal becomes so small that it is drowned out by the noise, causing the model to stall or diverge.

DP-NormFedAvg: A Normalization Approach

Instead of clipping, the proposed DP-NormFedAvg algorithm normalizes every client update. For a given update $u$, the client sends a scaled unit vector:

$$\text{update} = C \cdot \frac{u}{\|u\|}$$

This approach completely discards the magnitude of the update, retaining only the directional information. While it may seem counterintuitive to throw away data, this normalization ensures that the "signal" sent to the server always has a consistent magnitude $C$. This maintains a much healthier SNR throughout the entire training process, especially in the later stages of optimization.

Theoretical Breakthroughs

This work provides the first convergence analysis for private FL on smooth convex objectives using a general clipping/normalization threshold without the crutch of the Lipschitz assumption. Key theoretical insights include:

  1. Convergence Guarantees: The authors prove that normalization-based private FL can converge even when the underlying functions do not have bounded gradients.
  2. The Role of Local Steps: The analysis explores the impact of multiple local steps ($E$). While increasing $E$ can help the model overcome a poor initial starting point, it also interacts with data heterogeneity. If client datasets are highly diverse, too many local steps can actually hinder convergence in the private setting.
  3. SNR Superiority: Theoretically, normalization is shown to provide a more robust signal than clipping when noise is added, as it prevents the signal from shrinking relative to the noise floor.

Experimental Results

The researchers corroborated their theory through experiments on synthetic quadratic problems and standard benchmarking datasets: Fashion-MNIST (FMNIST), CIFAR-10, and CIFAR-100. For the CIFAR datasets, features were extracted using a pre-trained ResNet-18 to simulate realistic transfer learning scenarios.

Performance Gains

Across various privacy budgets ($\epsilon$), DP-NormFedAvg consistently outperformed the standard clipping-based DP-FedAvg:

  • CIFAR-100: At a privacy level of $\epsilon = 5$, normalization achieved an accuracy of 59.36%, compared to 56.53% for clipping—a significant improvement of nearly 3%.
  • Fashion-MNIST: In low-privacy regimes, normalization maintained a lead of over 2% in accuracy.
  • Optimization Trajectories: Synthetic tests revealed that DP-NormFedAvg followed a much more direct path to the true optimum. In contrast, clipping-based methods often wandered or stalled as the update magnitudes decreased and the noise became dominant.

Conclusion

DP-NormFedAvg represents a shift in how we think about privacy in federated learning. By demonstrating that directional information is often more valuable than magnitude—especially when that magnitude is obscured by DP noise—this work provides a more robust framework for private optimization. While the current theoretical results focus on convex objectives, the empirical success on ResNet features suggests broad applicability. Future research will look toward extending these formal guarantees to non-convex landscapes and exploring adaptive scaling factors to further optimize the convergence rate.

Summary updated

Abstract

There is a dearth of convergence results for differentially private federated learning (FL) with non-Lipschitz objective functions (i.e., when gradient norms are not bounded). The primary reason for this is that the clipping operation (i.e., projection onto an $\ell_2$ ball of a fixed radius called the clipping threshold) for bounding the sensitivity of the average update to each client's update introduces bias depending on the clipping threshold and the number of local steps in FL, and analyzing this is not easy. For Lipschitz functions, the Lipschitz constant serves as a trivial clipping threshold with zero bias. However, Lipschitzness does not hold in many practical settings; moreover, verifying it and computing the Lipschitz constant is hard. Thus, the choice of the clipping threshold is non-trivial and requires a lot of tuning in practice. In this paper, we provide the first convergence result for private FL on smooth \textit{convex} objectives \textit{for a general clipping threshold} -- \textit{without assuming Lipschitzness}. We also look at a simpler alternative to clipping (for bounding sensitivity) which is \textit{normalization} -- where we use only a scaled version of the unit vector along the client updates, completely discarding the magnitude information. {The resulting normalization-based private FL algorithm is theoretically shown to have better convergence than its clipping-based counterpart on smooth convex functions. We corroborate our theory with synthetic experiments as well as experiments on benchmarking datasets.
BibTeX
@misc{das2021dp,
  author = {Rudrajit Das and Abolfazl Hashemi and Sujay Sanghavi and Inderjit S. Dhillon},
  title = {{DP-NormFedAvg: Normalizing Client Updates for Privacy-Preserving Federated Learning}},
  year = {2021},
  eprint = {2106.07094},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.07094},
}
Figure