Factor-Assisted Federated Learning for Personalized Optimization with Heterogeneous Data

Feifei Wang, Huiyun Tang, Yang Li

TL;DR

This paper introduces FedSplit, a novel federated learning framework that addresses data heterogeneity by decomposing neural network layers into "shared" and "personalized" components. By leveraging factor analysis (FedFac), they effectively separate common knowledge from client-specific insights, leading to faster convergence and improved performance on heterogeneous datasets.

What Problem Does This Paper Address?

Federated Learning (FL) allows multiple clients to collaboratively train a global model without sharing their raw data, preserving privacy. However, a major hurdle is data heterogeneity, where data distributions across clients differ significantly. This non-IID (non-independent and identically distributed) nature can severely degrade the performance and convergence speed of standard FL algorithms like FedAvg. Existing solutions often focus on either adjusting global models or personalizing them, but this paper argues that data itself contains both shared and personalized knowledge, and this insight can be directly exploited.

Methods

The core innovation is the FedSplit framework. It hypothesizes that hidden layers in deep neural networks can be decomposed into:

  • Client-shared elements: Representing common knowledge across clients.
  • Client-specific elements: Capturing personalized knowledge unique to each client.

This decomposition allows for a novel objective function where shared parameters are updated collaboratively (similar to FedAvg), while personalized parameters are updated locally by each client.

To practically implement this decomposition, the authors introduce FedFac, which uses factor analysis. Factor analysis is a statistical technique used to identify underlying "latent factors" that explain the correlations between observed variables. In FedFac, it's applied to the hidden layer parameters (weights/gradients) to distinguish between shared and personalized components. They propose both static (one-time decomposition) and dynamic (re-evaluating decomposition each round) algorithms for FedFac.

The paper also provides theoretical analysis of FedSplit, guaranteeing linear convergence and generalization bounds, particularly under the neural tangent kernel (NTK) framework.

Datasets

The paper uses the following datasets for evaluation:

  • FEMNIST: Handwritten digit recognition, with data generated per writer.
  • Shakespeare: Next-character prediction, with data from different speaking roles.
  • CIFAR10 & CIFAR100: Image classification, with controlled levels of label heterogeneity using Dirichlet distributions and varying numbers of classes per client.

Key Insights and Ideas

  • Data inherently contains shared and personalized knowledge: This is the foundational insight. Instead of forcing a single global model or purely personalizing, FedSplit directly models this dual nature.
  • Decomposing neural network layers is key: The paper argues that hidden layer elements (neurons/channels) can be meaningfully split into shared and personalized groups, leading to more effective learning.
  • Factor analysis is a powerful tool for decomposition: FedFac demonstrates that factor analysis can effectively disentangle these shared and personalized components in a practical manner.
  • Dynamic decomposition offers benefits: The dynamic FedFac algorithm, which re-evaluates the decomposition over time, generally outperforms the static approach, suggesting that the nature of shared vs. personalized knowledge can evolve during training.
  • Personalized components are crucial for heterogeneous data: Ablation studies show that masking personalized parameters leads to a significant performance drop, especially in highly heterogeneous settings, highlighting their importance.

Key Results

  • Faster Convergence: FedSplit (and its practical implementation FedFac) demonstrates significantly faster convergence compared to standard FedAvg, both theoretically and empirically, especially under high data heterogeneity.
  • Improved Accuracy: FedFac consistently outperforms FedAvg, FedProx, and several other state-of-the-art personalized FL methods across various datasets and heterogeneity levels. For example, on CIFAR10 with π=0.1, FedFac achieves an accuracy of 0.8307, compared to FedAvg's 0.5722.
  • Robustness to Heterogeneity: FedFac shows robustness across different levels of data heterogeneity, maintaining strong performance even in challenging scenarios.
  • Effective Decomposition: Simulation studies show that factor analysis can accurately recover the true underlying decomposition of shared and personalized parameters.
  • Improved Fairness: FedFac leads to a smaller variance in local client accuracies, suggesting improved fairness in performance across clients.
  • Better New Client Prediction: FedFac with an "Ensemble" prediction strategy for new clients outperforms other methods.

Limitations and Future Work

  • Hyper-parameter Tuning: The paper acknowledges that choosing the optimal network layers for decomposition (parameter Λ), personalization threshold (τ), and variability threshold (κ) can be time-consuming. They suggest exploring automated hyper-parameter tuning strategies.
  • Privacy: While FedSplit is a federated learning method, it doesn't inherently incorporate advanced privacy-preserving techniques like differential privacy. The authors suggest this as a future research direction.
  • Theoretical Analysis for Dynamic FedFac: The current theoretical analysis primarily applies to the static FedFac method. Investigating the theoretical properties of the dynamic FedFac for decomposition is an open area.
  • Interpretability of Shared Filters: While personalized filters in FedFac extract clearer features, the shared filters in FedFac and FedAvg both generate more noise, suggesting room for improvement in understanding and utilizing shared representations.

Summary updated

Abstract

Federated learning is an emerging distributed machine learning framework aiming at protecting data privacy. Data heterogeneity is one of the core challenges in federated learning, which could severely degrade the convergence rate and prediction performance of deep neural networks. To address this issue, we develop a novel personalized federated learning framework for heterogeneous data, which we refer to as FedSplit. This modeling framework is motivated by the finding that, data in different clients contain both common knowledge and personalized knowledge. Then the hidden elements in each neural layer can be split into the shared and personalized groups. With this decomposition, a novel objective function is established and optimized. We demonstrate FedSplit enjoyers a faster convergence speed than the standard federated learning method both theoretically and empirically. The generalization bound of the FedSplit method is also studied. To practically implement the proposed method on real datasets, factor analysis is introduced to facilitate the decoupling of hidden elements. This leads to a practically implemented model for FedSplit and we further refer to as FedFac. We demonstrated by simulation studies that, using factor analysis can well recover the underlying shared/personalized decomposition. The superior prediction performance of FedFac is further verified empirically by comparison with various state-of-the-art federated learning methods on several real datasets.
BibTeX
@misc{wang2023factor,
  author = {Feifei Wang and Huiyun Tang and Yang Li},
  title = {{Factor-Assisted Federated Learning for Personalized Optimization with Heterogeneous Data}},
  year = {2023},
  eprint = {2312.04281},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.04281},
}
Figure