On Bridging Generic and Personalized Federated Learning for Image Classification

Hong-You Chen, Wei-Lun Chao

TL;DR

This paper introduces FED-ROD (Federated Robust Decoupling), a framework that simultaneously achieves state-of-the-art performance in both "generic" (global) and "personalized" (local) federated learning. By decoupling a model into two specialized prediction heads—one for global robustness and one for local accuracy—the authors resolve the long-standing dilemma of whether to prioritize the server's needs or the client's.

What Problem Does This Paper Address?

Federated Learning (FL) allows multiple clients (like mobile phones or hospitals) to collaboratively train a model without sharing their private data. However, a major hurdle is non-IID data: different clients have very different data distributions (e.g., one user takes photos of cats, another of cars).

The dilemma between Generic and Personalized FL performance.

This data divergence leads to a tug-of-war between two competing goals:

  1. Generic FL (G-FL): Aims for a single "global" model that performs well on the union of all data. This is useful for the server to deploy to new, unseen users.
  2. Personalized FL (P-FL): Aims to give each client a model tailored to their specific data.

Until now, the community has largely treated these as two separate tracks. Algorithms that excel at personalization often produce poor global models, and standard global algorithms (like FedAvg) often see a performance drop on individual clients when data is highly skewed.

Methods

The authors propose FED-ROD, which uses a "two-loss, two-predictor" architecture on top of a shared feature extractor to explicitly decouple the model's dual duties.

Comparison of standard ERM/BRM approaches with the proposed FED-ROD architectures.
  • Balanced Risk Minimization (BRM) for the Generic Head: To make the global model robust, the authors treat each client's local training as a class-imbalanced problem. They use a "balanced loss" (like Balanced Softmax) to ensure the model learns to classify every class well, even if a specific client only sees a few of them. This prevents the global model from being biased by local skews.
  • Empirical Risk for the Personalized Head: Simultaneously, a lightweight "P-head" is trained to minimize the client's actual empirical risk (their specific data distribution). This head acts as an "add-on" to the generic head, capturing the local nuances that the global model might ignore.
  • Hypernetworks for Adaptation: In one version, they use a hypernetwork to generate the personalized head based on a client's class distribution. This allows the system to provide a personalized model to a brand new client instantly (zero-shot) just by knowing what categories of data they have.

Datasets

The researchers evaluated FED-ROD on several standard image classification benchmarks, simulated under various "non-IID" settings (using Dirichlet distributions to create label skew):

  • CIFAR-10 and CIFAR-100: Standard object recognition.
  • Fashion-MNIST (FMNIST): Clothing items.
  • EMNIST: A large-scale dataset of handwritten letters and digits from thousands of different writers.

Key Insights and Ideas

  • Personalization "For Free": One of the most surprising findings is that standard FedAvg actually produces very strong personalized models if you simply keep the local checkpoints instead of discarding them.
  • Implicit Regularization: The authors argue that the act of "global aggregation" (averaging weights at the server) acts as a powerful regularizer. It prevents local models from over-fitting to their tiny, skewed datasets, which is why they often outperform dedicated P-FL algorithms.
  • The Power of Decoupling: You can't ask one prediction head to be both "unbiased" (for the server) and "biased toward local data" (for the client) at the same time. By splitting the heads, the feature extractor learns a rich, shared representation that serves both goals.

Key Results

  • Superior Performance: FED-ROD consistently outperformed both generic (FedAvg, FedProx, FedDyn) and personalized (Ditto, Per-FedAvg, pFedMe) baselines across all datasets.
  • Best of Both Worlds: In CIFAR-100 (Dir 0.3), FED-ROD achieved a generic accuracy of 48.5% (vs. FedAvg's 46.4%) and a personalized accuracy of 62.5% (vs. FedAvg's 61.7%).
  • Zero-Shot Success: The hypernetwork version showed it could generate highly effective personalized models for new clients who weren't even part of the initial training phase, significantly outperforming simple fine-tuning.
  • Compatibility: FED-ROD is "plug-and-play." When combined with other advanced FL optimizers like FedDyn, it pushed performance even higher.

Limitations and Future Work

  • Label Skew Focus: The paper primarily addresses "label skew" (different class distributions). It doesn't deeply explore "feature skew" (e.g., the same class appearing differently across clients due to lighting or camera quality).
  • Metadata Requirement: The hypernetwork and BRM components require knowing the client's class distribution (the count of samples per class). While this is less sensitive than raw data, it is still a form of metadata that must be shared with the server.
  • Future Directions: The authors suggest that designing even better losses for Balanced Risk Minimization in the federated context is a promising path forward.

Summary updated

Abstract

Federated learning is promising for its capability to collaboratively train models with multiple clients without accessing their data, but vulnerable when clients' data distributions diverge from each other. This divergence further leads to a dilemma: "Should we prioritize the learned model's generic performance (for future use at the server) or its personalized performance (for each client)?" These two, seemingly competing goals have divided the community to focus on one or the other, yet in this paper we show that it is possible to approach both at the same time. Concretely, we propose a novel federated learning framework that explicitly decouples a model's dual duties with two prediction tasks. On the one hand, we introduce a family of losses that are robust to non-identical class distributions, enabling clients to train a generic predictor with a consistent objective across them. On the other hand, we formulate the personalized predictor as a lightweight adaptive module that is learned to minimize each client's empirical risk on top of the generic predictor. With this two-loss, two-predictor framework which we name Federated Robust Decoupling (Fed-RoD), the learned model can simultaneously achieve state-of-the-art generic and personalized performance, essentially bridging the two tasks.
BibTeX
@inproceedings{chen2022bridging,
  author = {Hong-You Chen and Wei-Lun Chao},
  title = {{On Bridging Generic and Personalized Federated Learning for Image Classification}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=QuBYtHLGho},
}
Figure