FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization

Fan Zhang, Carlos Esteve Yague, Sören Dittmer, Carola-Bibiane Schönlieb, Michael robertsSubmission to ICLR 2025 · Rejected

TL;DR

FedMAP is a personalized federated learning framework that uses Maximum A Posteriori (MAP) estimation to tackle real-world data heterogeneity across decentralized clients. Instead of averaging models directly to fit a single consensus, the central server iteratively optimizes a shared parametric prior distribution that acts as a regularizer for local client models. This bi-level formulation significantly outperforms standard baselines across diverse forms of non-IID data skews.

What Problem Does This Paper Address?

Standard Federated Learning (FL) algorithms like FedAvg rely on Maximum Likelihood Estimation (MLE), training a single global model to minimize the average loss across all participating devices. In practice, however, client data is rarely identically distributed (non-IID). Clients often encounter combinations of:

  1. Quantity skew (varying sample counts),
  2. Feature distribution skew (different measurement conditions or inputs), and
  3. Label distribution skew (different class proportions).

When local distributions diverge significantly, gradients from different clients pull in opposite directions (client drift), leading to poor convergence or a consensus model that performs badly for everyone. Personalized Federated Learning (PFL) attempts to balance local adaptation with global collaboration, but existing methods often struggle when multiple non-IID factors occur simultaneously.

Methods

FedMAP casts personalized FL as a bi-level MAP optimization problem:

  1. Local Optimization via MAP Estimation: Instead of minimizing just the local empirical risk $\mathcal{L}(\theta; \mathcal{Z}_k)$, each client $k$ incorporates a shared global prior distribution $\rho_\gamma(\theta)$ over parameters. Using a Gaussian prior with mean $\gamma$ and variance $\sigma^2$, the local MAP loss becomes: $$\min_\theta \mathcal{L}(\theta; \mathcal{Z}_k) + \frac{1}{2\sigma^2} \|\theta - \gamma^{(t)}\|^2$$ This naturally regularizes the local model towards the global parameter $\gamma^{(t)}$ without overwriting client-specific specializations.

  2. Server Aggregation as Prior Estimation: At each round, clients compute an importance weight reflecting their posterior likelihood: $$\omega_k^{(t)} = \mathbb{P}(\mathcal{Z}_k \mid \theta_k^{(t+1)}) \times \rho_{\gamma^{(t)}}(\theta_k^{(t+1)})$$ The server then performs gradient descent on the global prior parameters, which reduces to an adaptive weighted average: $$\gamma^{(t+1)} = \frac{1}{\sum_{j=1}^q \omega_j^{(t)}} \sum_{k=1}^q \omega_k^{(t)} \theta_k^{(t+1)}$$

  3. Theoretical Formulation: The authors prove that the alternating updates correspond to gradient descent over an envelope function $M(\gamma) = \sum_k w_k M_k(\gamma; \mathcal{Z}_k)$ (an anisotropic Moreau envelope when using Gaussian priors), guaranteeing convergence to the unique solution of a bi-level optimization objective under convexity assumptions.

Datasets

The authors evaluated FedMAP using synthetic and real-world benchmarks:

  • Synthetic Benchmark: A newly constructed 30-feature binary classification task with intrinsic dimension $d=4$, specifically designed to evaluate affine feature shifts, sample size disparities (e.g., 2000 vs. 500 samples), and extreme label imbalance (85% majority class).
  • Office-31: A real-world domain adaptation benchmark comprising 4,110 images across 31 classes from three visual domains: Amazon, Webcam, and DSLR.
  • FEMNIST (LEAF framework): A standard FL benchmark with 62 classes of handwritten characters across 36 non-IID client partitions.

Key Insights and Ideas

  • Optimizing an Envelope Function vs. Loss Averages: Standard FL tries to minimize the average loss $F(\theta) = \sum_k w_k \mathcal{L}(\theta; \mathcal{Z}_k)$. In heterogeneous setups, the minimizer of $F(\theta)$ can fail disastrously (e.g., producing positive slopes for data generated with negative slopes). In contrast, minimizing the combination of envelope functions $M(\gamma)$ properly preserves local trends while learning global structure.
  • Prior-Based Adaptive Weighting: Typical FL methods weight client updates strictly by sample size $N_k$. FedMAP weights clients by model confidence and relevance using posterior likelihood, protecting the global prior from being skewed by noisy or misaligned updates.
  • Disadvantaged Clients Benefit Most: Clients with severe class imbalance or limited samples gain the highest accuracy boosts by borrowing global prior knowledge from better-represented clients.

Key Results

FedMAP was benchmarked against standard baselines (FedAvg, FedProx, FedBN) and Individual Training:

  • Synthetic Label Skew: Clients with severely skewed label distributions improved dramatically. FedMAP achieved an overall average accuracy of 84.84% compared to 76.63% (Individual), 63.28% (FedBN), 54.91% (FedProx), and 54.66% (FedAvg). Highly imbalanced clients saw single-client jumps of over 19% compared to local training.
  • Synthetic Quantity & Feature Skew: FedMAP reached 79.06% on quantity skew (vs. 75.37% individual, ~61–63% standard baselines) and 86.27% on feature skew (vs. 83.38% individual, ~63–65% baselines).
  • Office-31 Benchmark: FedMAP achieved an average accuracy of 83.97%, beating Individual training (73.45%), FedBN (78.93%), FedProx (60.04%), and FedAvg (61.52%). The low-resolution Webcam client improved by +17.06% over local training by leveraging higher-resolution representations from DSLR.
  • FEMNIST: FedMAP averaged 84.15% validation accuracy across 36 clients, surpassing the next best baseline (FedProx at 79.37%) and individual training (78.64%).

Limitations and Future Work

  • Isotropic Gaussian Assumption: The current formulation assumes an isotropic Gaussian distribution for the parameter prior, using the mean as the primary learned parameter.
  • Covariance Matrix Tuning: The variance parameter $\sigma^2$ was fixed rather than dynamically adapted per-layer or per-feature during training.
  • Future Directions: Exploring alternative, non-Gaussian prior distributions and learning full parameter covariance matrices during federated aggregation.

Summary updated

Abstract

Federated Learning (FL) enables collaborative training of machine learning (ML) models on decentralized data while preserving data privacy. However, data across clients often differs significantly due to class imbalance, feature distribution skew, sample size imbalance, and other phenomena. Using information from these not identically distributed (non-IID) datasets causes challenges in training. Existing FL methods based on a single global model cannot effectively capture client data variations, resulting in suboptimal performance. Personalized FL (PFL) techniques were introduced to adapt to the local data distribution of each client and utilize the data from other clients. They have shown promising results in addressing these challenges. We propose FedMAP, a novel Bayesian PFL framework which applies Maximum A Posteriori (MAP) estimation to effectively mitigate various non-IID data issues, by means of a parametric prior distribution, which is updated during aggregation. We provide a theoretical foundation illustrating FedMAP's convergence properties. In particular, we prove that the prior updates in FedMAP correspond to gradient descent iterations for a linear combination of envelope functions associated with the local losses. This differs from previous FL approaches, that aim at minimizing a weighted average of local loss functions and often face challenges with heterogeneous data distributions, resulting in reduced client performance and slower convergence in non-IID settings. Finally, we show, through evaluations of synthetic and real-world datasets, that FedMAP achieves better performance than the existing methods. Moreover, we offer a robust, ready-to-use framework to facilitate practical deployment and further research.
BibTeX
@misc{zhang2024fedmap,
  author = {Fan Zhang and Carlos Esteve Yague and Sören Dittmer and Carola-Bibiane Schönlieb and Michael roberts},
  title = {{FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization}},
  year = {2024},
  eprint = {2405.19000},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.19000},
}
Figure