Abstract
Federated Learning (FL) often faces a trade-off between global generalization and local personalization. While Personalized FL (PFL) improves local performance, it frequently leads to "over-personalization," where models fall into sharp minima and fail to generalize to out-of-distribution (OOD) data. We present FedSoup, a framework that leverages selective model interpolation ("souping") of historical global models and local updates. FedSoup steers models toward flatter regions of the loss landscape, enhancing both local accuracy and robustness to unseen domains. Evaluated on medical imaging tasks, FedSoup consistently outperforms state-of-the-art FL methods.
Introduction
In Federated Learning, institutions like hospitals collaborate to train models without sharing sensitive data. A major hurdle is the "distribution shift"—variations in data due to different equipment, protocols, or patient demographics. Standard Global FL aims for a single universal model but often yields suboptimal local performance. Conversely, Personalized FL (PFL) tailors models to specific clients, which can lead to "sharp" local minima. These models overfit to local nuances, making them fragile when encountering new, unseen data distributions.
Methodology: The FedSoup Framework
FedSoup addresses the personalization-generalization gap through two primary mechanisms: Temporal Model Selection and Federated Model Patching.
Temporal Model Selection
Instead of the computationally expensive approach of training multiple models with varying hyperparameters, FedSoup utilizes the temporal history of the global model. Global models from different communication rounds serve as "ingredients" for the soup. Each client employs a greedy selection strategy, maintaining a "soup" of historical global models that collectively improve performance on a local validation set. This approach utilizes existing training artifacts without additional communication overhead.
Federated Model Patching
To prevent local models from drifting into sharp minima, FedSoup interpolates the current local model with the selected global model soup. This "patching" process bridges the gap between local and global domains. By averaging the weights of $k$ selected global models ($ heta_g^i$) and the current local model ($ heta_l$), the final model is steered toward a "flat minimum"—a region in the loss landscape where performance is more robust to parameter or data perturbations.
The update rule is defined as: $$\theta_{FedSoup} \leftarrow \frac{\sum_{i=1}^{k} \theta_g^i + \theta_l}{k + 1}$$
Experimental Evaluation
FedSoup was evaluated on two medical image classification datasets characterized by significant domain shifts:
- Camelyon17 (Pathology): 4,600 images from five sources for tumor identification in lymph node slides.
- Retinal Fundus Images: 1,264 images from four institutions for identifying abnormal retinal conditions.
Key Results and Analysis
FedSoup demonstrates superior performance across both local and global metrics compared to baselines like FedAvg, FedProx, and FedRep.
Generalization and Sharpness
A critical finding is the correlation between model generalization and the geometry of the loss landscape. Sharpness quantification using Hessian eigenvalues reveals that FedSoup models reside in significantly flatter regions compared to other PFL methods. This flatness directly translates to better performance on unseen domains. In "leave-one-out" cross-institution tests, FedSoup achieved a 2.87-point increase in AUC over FedAvg.
Performance Summary
On the Camelyon17 pathology dataset, FedSoup achieved a global accuracy of 72.87%, outperforming FedAvg (70.18%). For Retinal Fundus images, it reached a global AUC of 86.24%, surpassing FedBABU by over 1%. These results confirm that selective interpolation effectively balances local needs with global robustness.
Conclusion
FedSoup provides a practical and efficient solution for the distribution shift problem in Federated Learning. By repurposing historical global models and employing selective interpolation, it achieves flatter minima and superior generalization. Future work will investigate the sensitivity of the interpolation start epoch and the framework's performance in non-image domains like NLP.
Summary updated