A federated learning differential privacy algorithm for non-Gaussian heterogeneous data

Xinyu Yang, Weisan Wu

Abstract

Multi-center heterogeneous data are a hot topic in federated learning. The data of clients and centers do not follow a normal distribution, posing significant challenges to learning. Based on the assumption that the client data have a multivariate skewed normal distribution, we improve the DP-Fed-mv-PPCA model. We use a Bayesian framework to construct prior distributions of local parameters and use expectation maximization and pseudo-Newton algorithms to obtain robust parameter estimates. Then, the clipping algorithm and differential privacy algorithm are used to solve the problem in which the model parameters do not have a display solution and achieve privacy guarantee. Furthermore, we verified the effectiveness of our model using synthetic and actual data from the Internet of vehicles.

Introduction

Standard Federated Learning (FL) and Differential Privacy (DP) frameworks often rely on the assumption that data follows a Gaussian (normal) distribution. However, real-world high-dimensional data, such as vehicle trajectory networks in the Internet of Vehicles (IoV), are frequently heterogeneous, asymmetric, or heavy-tailed. When models assuming normality encounter such "irregular" data, estimation accuracy degrades significantly. Furthermore, while FL keeps raw data local, training parameters can still leak sensitive information.

This paper introduces DFmP-SN, a federated learning algorithm designed to handle non-Gaussian heterogeneous data. By replacing standard Gaussian assumptions with a multivariate skewed normal distribution and integrating differential privacy, the proposed method achieves higher model accuracy and stronger privacy guarantees for complex datasets.

Methodology

The DFmP-SN model improves upon the Federated Multi-View Probabilistic Principal Component Analysis (DP-Fed-mv-PPCA) framework. The approach integrates several technical layers to handle data skewness and privacy:

  1. Skewed Normal Modeling: The model utilizes a multivariate skewed normal distribution instead of standard Gaussian noise. This introduces a "skewness parameter" that allows the model to flexibly describe data asymmetry.
  2. Bayesian Framework: Prior distributions for local parameters are constructed to manage missing views and statistical heterogeneity across different service centers.
  3. Robust Parameter Estimation: The model employs Expectation Maximization (EM) combined with pseudo-Newton algorithms to iteratively derive robust parameter estimates.
The multi-view federated learning architecture for heterogeneous data centers.
  1. Privacy Mechanism: To ensure client-level privacy, a clipping algorithm bounds the sensitivity of parameters. Noise is then injected using Gaussian and Laplace mechanisms to satisfy $(\epsilon, \delta)$-differential privacy standards.

Experimental Evaluation

The model was evaluated using two datasets:

  • Synthetic Data (SD): 10,000 observations across three views with varying dimensions to test performance under controlled non-Gaussian conditions.
  • Didi Chuxing GAIA Data (DiDi): A real-world dataset of 10,000 vehicle trajectory points from Chengdu, China. Longitude and latitude were converted to east-north coordinates to represent complex traffic patterns.

Results and Discussion

The DFmP-SN model demonstrates superior performance in handling non-Gaussian data compared to baseline methods like FedVCP and standard Fed-mv-PPCA.

Comparison of Mean Squared Error (MSE) across iterations for DFmP-SN and baseline models.

As shown in the performance analysis, DFmP-SN consistently achieves a lower Mean Squared Error (MSE) across iterations compared to traditional Gaussian-based models, particularly as the dimensionality ($d$) increases. The "Aha" moment for this research was the realization that the "tails" of a distribution in vehicle networking represent critical patterns rather than outliers. By modeling skewness directly, the algorithm treats irregular data as a feature.

Furthermore, the model maintains a favorable privacy-utility balance. Even with a tightened privacy budget ($\epsilon$), the skewed normal approach maintains a significantly lower error rate than Gaussian-based models. Statistical measures like AIC and BIC further confirmed that the non-Gaussian fit was more effective for real-world traffic data.

Conclusion and Future Work

DFmP-SN effectively addresses the challenges of non-Gaussian heterogeneous data in federated learning. While the model shows promise, future work will focus on reducing the computational complexity of DP calculations for large-scale heterogeneous sources. Additionally, the authors plan to extend the framework to handle matrix and tensor data structures and investigate the Huber contamination model to further enhance robustness against extreme outliers.

Summary updated

Abstract

Multi-center heterogeneous data are a hot topic in federated learning. The data of clients and centers do not follow a normal distribution, posing significant challenges to learning. Based on the assumption that the client data have a multivariate skewed normal distribution, we improve the DP-Fed-mv-PPCA model. We use a Bayesian framework to construct prior distributions of local parameters and use expectation maximization and pseudo-Newton algorithms to obtain robust parameter estimates. Then, the clipping algorithm and differential privacy algorithm are used to solve the problem in which the model parameters do not have a display solution and achieve privacy guarantee. Furthermore, we verified the effectiveness of our model using synthetic and actual data from the Internet of vehicles.
BibTeX
@inproceedings{yang2023federated,
  author = {Xinyu Yang and Weisan Wu},
  title = {{A federated learning differential privacy algorithm for non-Gaussian heterogeneous data}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-33044-y},
}
Figure