Dynamic Personalized Federated Learning with Adaptive Differential Privacy

Xiyuan Yang, Wenke Huang, Mang Ye

Personalized Federated Learning (PFL) combined with Differential Privacy (DP) is a powerful paradigm for training models on decentralized, non-IID (non-independent and identically distributed) data while strictly protecting user privacy. However, current methods often suffer from degraded performance due to inflexible model partitioning and the destructive effects of gradient clipping.

To address these bottlenecks, researchers introduce FedDPA (Federated learning with Dynamic Fisher Personalization and Adaptive Constraint). This novel framework dynamically selects which parameters to personalize—shielding them from DP noise—and applies adaptive constraints to smooth out the distortions caused by gradient clipping. The result is a more flexible, robust, and accurate federated learning system under strict privacy budgets.

The Challenge: DP in Personalized Federated Learning

Federated Learning allows multiple clients to train a shared model without exposing their raw data. Because real-world client data is rarely identically distributed, researchers use PFL to tailor specific parts of the model to individual clients. To prevent inference attacks from reconstructing client data, User-Level Differential Privacy is applied, relying on two main operations: gradient clipping (bounding the size of updates) and noise addition.

Combining DP and PFL introduces two major bottlenecks:

  1. Inflexible Personalization & Noise Vulnerability: Most PFL methods statically partition the model (e.g., always personalizing the final classifier layer). This rigid approach ignores diverse client data characteristics. Furthermore, standard DP blindly adds noise to all parameters, which can overwrite and destroy the most informative local weights.
  2. Gradient Distortion: DP requires clipping local updates to a maximum $L_2$ norm. Traditional optimization methods ignore this hard constraint during local training. When updates inevitably hit the clipping bound, the gradients are harshly truncated and distorted, leading to agonizingly slow convergence or overall model degradation.

The Solution: FedDPA

To tackle these issues, FedDPA introduces two key mechanisms that rethink how parameters are partitioned and regularized:

1. Dynamic Fisher Personalization (DFP)

Instead of a fixed parameter split, FedDPA dynamically determines which parameters to personalize for each client before local training begins. It calculates the layer-wise Fisher Information Matrix (FIM) to estimate the "information content" of each parameter.

Parameters with high Fisher values are deemed highly sensitive and informative. These are masked, retained locally, and protected from being overwritten by noisy global updates. Less informative parameters are updated and shared with the global model. By using Fisher Information as a "privacy shield," the model preserves local utility without violating the mathematical guarantees of DP.

2. Adaptive Constraint (AC)

To combat clipping distortion, FedDPA abandons uniform $L_2$ regularization. Instead, it applies two distinct constraints during local training:

  • For personalized parameters: It applies standard regularization to retain local knowledge.
  • For shared parameters: It introduces a bounded regularization term that actively pushes the $L_2$ norm of the updates to closely approach the DP clipping bound.

By integrating the clipping bound directly into the local loss function, the model learns to "gracefully brake" before hitting the clipping wall. This pre-emptive mitigation preserves the directional integrity of the gradient.

Experimental Setup & Key Results

The authors evaluated FedDPA on three standard image classification datasets adapted for federated, non-IID settings: FEMNIST (inherently partitioned by writer), CIFAR-10 (partitioned using a Dirichlet distribution to simulate varying non-IID degrees), and SVHN (real-world digit classification).

FedDPA consistently outperformed state-of-the-art DP-PFL methods—including DP-FedAvg, BLUR+LUS, PPSGD, and DP-FedSAM—across various privacy budgets ($\epsilon$) and non-IID settings:

  • FEMNIST: At a privacy budget of $\epsilon=4$, FedDPA achieved 77.27% accuracy, outperforming the next best method (BLUR+LUS at 74.23%) by over 3%.
  • SVHN: At $\epsilon=4$, FedDPA reached 62.63% accuracy, delivering a massive ~5% absolute improvement over BLUR+LUS (57.52%).
  • CIFAR-10 (High Non-IID): Under highly heterogeneous data distributions ($\alpha=1$), FedDPA maintained 49.75% accuracy, beating BLUR+LUS (48.97%) and DP-FedSAM (45.07%).
  • Ablation Studies: Both components are vital. Removing either the Dynamic Fisher Personalization or the Adaptive Constraint resulted in a 2–5% drop in accuracy across all privacy budgets.

Limitations and Future Directions

While FedDPA represents a significant step forward, it does have limitations. The method currently assumes model homogeneity, meaning all clients must share the exact same model architecture. It cannot natively handle heterogeneous model structures, which is a common reality in cross-device FL. Additionally, calculating the Fisher information and generating parameter masks introduces a computational overhead per global epoch, though the authors note this is an order of magnitude smaller than the actual training expense.

Future work could extend the core idea of using Fisher information to selectively share parameters for feature selection in high-dimensional datasets, or adapt the framework to support heterogeneous model architectures across diverse client devices.

Summary updated

Abstract

Personalized federated learning with differential privacy has been considered a feasible solution to address non-IID distribution of data and privacy leakage risks. However, current personalized federated learning methods suffer from inflexible personalization and convergence difficulties due to two main factors: 1) Firstly, we observe that the prevailing personalization methods mainly achieve this by personalizing a fixed portion of the model, which lacks flexibility. 2) Moreover, we further demonstrate that the default gradient calculation is sensitive to the widely-used clipping operations in differential privacy, resulting in difficulties in convergence. Considering that Fisher information values can serve as an effective measure for estimating the information content of parameters by reflecting the model sensitivity to parameters, we aim to leverage this property to address the aforementioned challenges. In this paper, we propose a novel federated learning method with Dynamic Fisher Personalization and Adaptive Constraint (FedDPA) to handle these challenges. Firstly, by using layer-wise Fisher information to measure the information content of local parameters, we retain local parameters with high Fisher values during the personalization process, which are considered informative, simultaneously prevent these parameters from noise perturbation. Secondly, we introduce an adaptive approach by applying differential constraint strategies to personalized parameters and shared parameters identified in the previous for better convergence. Our method boosts performance through flexible personalization while mitigating the slow convergence caused by clipping operations. Experimental results on CIFAR-10, FEMNIST and SVHN dataset demonstrate the effectiveness of our approach in achieving better performance and robustness against clipping, under personalized federated learning with differential privacy.
BibTeX
@inproceedings{yang2023dynamic,
  author = {Xiyuan Yang and Wenke Huang and Mang Ye},
  title = {{Dynamic Personalized Federated Learning with Adaptive Differential Privacy}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=RteNLuc8D9},
}
Figure