From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matching

Nannan Wu, Zhuo Kuang, Zengqiang Yan, Li YuIJCAI 2024

Introduction

Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models across decentralized institutions, such as hospitals, without compromising data privacy. However, a significant yet often overlooked hurdle in medical FL is Quality Shift. While institutional data may vary in labels or domains, the physical quality of imaging data—affected by hardware age, maintenance, or patient movement—can create severe disparities in model performance.

The Challenge of Quality Shift

In real-world medical settings, imaging quality is rarely uniform. A hospital with state-of-the-art equipment produces "clean" data, whereas another might deal with sensor noise or motion blur from uncooperative patients. Standard FL algorithms, like Federated Averaging (FedAvg), tend to optimize for the majority "clean" clients, leading to poor performance on the "corrupted" minority.

Illustration of quality shift in medical federated learning, where some clients possess corrupted data (noise or motion blur) while others have clean data.

Existing "fair" FL methods attempt to address this by reweighting clients based on training loss—giving more importance to those with higher errors. However, this paper argues that this approach is a "fairness trap." Forcing a model to fit noisy data often results in sharp minima: points where training loss is low, but the model fails to generalize to new, unseen testing data.

From Optimization to Generalization

The core innovation of this research is the shift from "fair optimization" to "fair generalization." Instead of merely balancing training losses, the authors propose balancing the "flatness" or "sharpness" of the loss landscape across all clients.

Comparison between fair optimization (minimizing training loss) and fair generalization (balancing the flatness of the loss landscape) for clean and corrupted clients.

Flat minima are mathematically linked to better generalization. If the model parameters shift slightly during deployment, a flat region ensures the loss remains stable, whereas a sharp region causes performance to plummet. By matching sharpness levels, the framework ensures that all clients—regardless of data quality—achieve equitable generalization performance.

The FedISM Framework

The proposed framework, FedISM (Federated learning with Inter-client Sharpness Matching), introduces two key components to the standard pipeline:

  1. Sharpness-Aware Local Training (SALT): Local clients utilize a technique inspired by Sharpness-Aware Minimization (SAM). Instead of just minimizing standard loss, SALT forces the model to find flat regions in the loss landscape.
  2. Sharpness-Aware Global Aggregation (SAGA): The central server aggregates model updates by weighting clients based on their sharpness levels. Clients with higher sharpness (typically those with corrupted data) receive higher weights. This process "matches" the sharpness across the network, harmonizing performance.

Importantly, FedISM remains privacy-preserving, as clients only need to share a single scalar value representing their sharpness level with the server.

Experimental Evaluation

The researchers evaluated FedISM on two large-scale medical imaging benchmarks: RSNA ICH (25,000 brain CT scans) and ISIC 2019 (25,000 skin disease images). To simulate quality shifts, 20% of the clients were assigned corrupted data (Gaussian noise or motion blur).

FedISM demonstrated significant improvements over state-of-the-art methods like q-FedAvg, FairFed, and FedGA:

  • Accuracy Boost: On the ICH dataset, FedISM outperformed standard FedAvg by 10.85% on corrupted images and surpassed the best existing fair FL method (FedGA) by 3.78%.
  • No "Fairness Tax": Unlike many fair algorithms that degrade performance on "strong" (clean) clients to help "weak" ones, FedISM maintained top-tier results on clean clients while simultaneously boosting the corrupted ones.
  • Robustness: The method remained effective across varying ratios of corrupted clients (10% to 30%), proving its resilience to different levels of institutional disparity.

Conclusion

FedISM represents a paradigm shift in fair federated learning. By moving beyond simple loss-balancing to the geometric matching of loss landscapes, it provides a robust solution for the inherent quality shifts in medical imaging. While sharpness-aware training doubles local computation time, the gains in generalization and fairness make it a compelling framework for deploying equitable AI in diverse healthcare environments.

Summary updated

BibTeX
@inproceedings{wu2024from,
  author = {Nannan Wu and Zhuo Kuang and Zengqiang Yan and Li Yu},
  title = {{From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matching}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure