Proximal Regularization for Personalized Federated Learning

This direction is split into other directions: regularized model personalization. Its last version stays readable and citable here.

Version 1 of 150 papers57 conceptsfrozen 2026-10-04

In standard Federated Learning (FL), the goal is to train a single "global" model that performs well across all participating devices. However, this "one-size-fits-all" approach often fails in the real world due to data heterogeneity. If one user primarily types in English and another in French, a global average model for predictive text will likely be mediocre for both.

To solve this, researchers have converged on a framework that treats the global model not as the final product, but as a shared foundation. By using proximal regularization, each participant can develop a personalized model that stays "close" to the global knowledge while adapting to local nuances.

The Elastic Leash: Balancing Global and Local

The core intuition behind this research direction is to treat personalization as a multi-task learning problem. Instead of forcing every node to reach a consensus, each node $i$ maintains its own local parameters $w_i$. The optimization objective for each node becomes a balancing act:

$$\min_{w_i} F_i(w_i) + \frac{\lambda}{2} \|w_i - w_g\|^2$$

Here, $F_i$ is the local loss on the user's data, and $w_g$ is the global reference model. The second term is a proximal regularizer—an "elastic leash" that penalizes the local model for wandering too far from the global consensus.

As Tian Li, Virginia Smith, and their colleagues demonstrated with the Ditto framework, this simple addition solves a major tension. By adjusting the regularization parameter $\lambda$, the system can navigate the trade-off between global generalization and local specificity. This mechanism doesn't just improve accuracy; it also enhances robustness. If a malicious actor tries to poison the global model, the proximal term allows honest nodes to ignore the corrupted global update in favor of their own local data.

Decoupling via Moreau Envelopes

While the "elastic leash" is intuitive, it couples the local and global updates tightly, which can slow down convergence. To address this, Canh T. Dinh and Nguyen Hoang Tran introduced pFedMe, which utilizes Moreau envelopes.

The Moreau envelope acts as a smoothing operator. In pFedMe, the global model is updated based on the "envelope" of the local functions rather than the raw local parameters. This effectively decouples the optimization: the inner loop optimizes the personalized model for the client, while the outer loop updates the global reference. This approach, also explored in the FLAME framework by Shengkun Zhu and Yuan Yao, allows for faster convergence and better handling of non-i.i.d. data without the computational burden of second-order derivatives.

Personalization as a Bayesian Prior

Another way to view this regularization is through the lens of Bayesian inference. In this framing, the global model is a prior distribution of what a "good" model looks like, and the local training is the process of finding the posterior.

Mingjia Shi and Jiancheng Lv 0001 proposed pFedBreD, which replaces standard L2 distance with Bregman divergence. This allows for more flexible "distances" between models, which is particularly useful in complex tasks like Knowledge Graph embedding, as seen in the work of Xiaoxiong Zhang on Client-Wise Relation Graphs. By treating the global model as a soft guide rather than a rigid template, these methods allow local models to retain specialized features that would otherwise be washed out by the global average.

Communication Efficiency and Minimax Guarantees

A common critique of personalized FL is that it might require more communication to keep all these models in sync. However, recent theoretical work suggests the opposite. Runze Li, Lingzhou Xue, and Xin Yu have provided minimax-optimal statistical rates for regularization-based PFL.

Their algorithm, FedCLUP, demonstrates that by tuning the degree of personalization, one can actually reduce communication costs. Because the local models are allowed to diverge slightly, the global model doesn't need to be updated as frequently to maintain high local accuracy. This is pushed to the extreme in Compressed L2GD by Peter Richtárik, which achieves massive data reductions by combining personalization with bidirectional communication compression.

Why It Works: The Convergence of Diverse Needs

The success of proximal regularization in this field stems from its ability to solve three problems at once:

  1. Accuracy: It allows models to adapt to local data distributions (non-i.i.d. data).
  2. Robustness: It prevents a single malicious or "outlier" node from dragging the entire network down, as noted by Sadegh Farhadkhani.
  3. Fairness: By personalizing constraints, frameworks like pFedFair ensure that the model performs well for all groups, not just the majority.

By moving away from the search for a single "perfect" model and instead building a "perfectly adaptable" system, this research direction provides a practical path for deploying AI in the messy, heterogeneous reality of edge devices.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_proximal_regularization_for_personalized_db82ccbd,
  title = {Proximal Regularization for Personalized Federated Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/proximal-regularization-for-personalized-federated-learning/rdr_db82ccbdbd6d4d30acd0e5f0fd75a144},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_proximal_regularization_for_personalized_db82ccbd_v1,
  title = {Proximal Regularization for Personalized Federated Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/proximal-regularization-for-personalized-federated-learning/rdr_db82ccbdbd6d4d30acd0e5f0fd75a144/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}