FedSoup: Improving Generalization and Personalization in Federated Learning via Selective Model Interpolation

Minghui Chen, Meirui Jiang, Qi Dou, Zehua Wang, Xiaoxiao Li

Abstract

Federated Learning (FL) often faces a trade-off between global generalization and local personalization. While Personalized FL (PFL) improves local performance, it frequently leads to "over-personalization," where models fall into sharp minima and fail to generalize to out-of-distribution (OOD) data. We present FedSoup, a framework that leverages selective model interpolation ("souping") of historical global models and local updates. FedSoup steers models toward flatter regions of the loss landscape, enhancing both local accuracy and robustness to unseen domains. Evaluated on medical imaging tasks, FedSoup consistently outperforms state-of-the-art FL methods.

Introduction

In Federated Learning, institutions like hospitals collaborate to train models without sharing sensitive data. A major hurdle is the "distribution shift"—variations in data due to different equipment, protocols, or patient demographics. Standard Global FL aims for a single universal model but often yields suboptimal local performance. Conversely, Personalized FL (PFL) tailors models to specific clients, which can lead to "sharp" local minima. These models overfit to local nuances, making them fragile when encountering new, unseen data distributions.

Methodology: The FedSoup Framework

FedSoup addresses the personalization-generalization gap through two primary mechanisms: Temporal Model Selection and Federated Model Patching.

Temporal Model Selection

Instead of the computationally expensive approach of training multiple models with varying hyperparameters, FedSoup utilizes the temporal history of the global model. Global models from different communication rounds serve as "ingredients" for the soup. Each client employs a greedy selection strategy, maintaining a "soup" of historical global models that collectively improve performance on a local validation set. This approach utilizes existing training artifacts without additional communication overhead.

Federated Model Patching

To prevent local models from drifting into sharp minima, FedSoup interpolates the current local model with the selected global model soup. This "patching" process bridges the gap between local and global domains. By averaging the weights of $k$ selected global models ($ heta_g^i$) and the current local model ($ heta_l$), the final model is steered toward a "flat minimum"—a region in the loss landscape where performance is more robust to parameter or data perturbations.

Conceptual comparison between common PFL and FedSoup, highlighting the transition from sharp to flat minima.

The update rule is defined as: $$\theta_{FedSoup} \leftarrow \frac{\sum_{i=1}^{k} \theta_g^i + \theta_l}{k + 1}$$

Experimental Evaluation

FedSoup was evaluated on two medical image classification datasets characterized by significant domain shifts:

  1. Camelyon17 (Pathology): 4,600 images from five sources for tumor identification in lymph node slides.
  2. Retinal Fundus Images: 1,264 images from four institutions for identifying abnormal retinal conditions.

Key Results and Analysis

FedSoup demonstrates superior performance across both local and global metrics compared to baselines like FedAvg, FedProx, and FedRep.

Generalization and Sharpness

A critical finding is the correlation between model generalization and the geometry of the loss landscape. Sharpness quantification using Hessian eigenvalues reveals that FedSoup models reside in significantly flatter regions compared to other PFL methods. This flatness directly translates to better performance on unseen domains. In "leave-one-out" cross-institution tests, FedSoup achieved a 2.87-point increase in AUC over FedAvg.

Quantitative analysis of FedSoup: (a) Sharpness quantification via Hessian eigenvalues, (b) Trade-off between ID and OOD accuracy, and (c) Performance on unseen domain generalization.

Performance Summary

On the Camelyon17 pathology dataset, FedSoup achieved a global accuracy of 72.87%, outperforming FedAvg (70.18%). For Retinal Fundus images, it reached a global AUC of 86.24%, surpassing FedBABU by over 1%. These results confirm that selective interpolation effectively balances local needs with global robustness.

Conclusion

FedSoup provides a practical and efficient solution for the distribution shift problem in Federated Learning. By repurposing historical global models and employing selective interpolation, it achieves flatter minima and superior generalization. Future work will investigate the sensitivity of the interpolation start epoch and the framework's performance in non-image domains like NLP.

Summary updated

BibTeX
@inproceedings{chen2023fedsoup,
  author = {Minghui Chen and Meirui Jiang and Qi Dou and Zehua Wang and Xiaoxiao Li},
  title = {{FedSoup: Improving Generalization and Personalization in Federated Learning via Selective Model Interpolation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  url = {https://openreview.net/forum?id=7gk8acU3kt},
}
Figure