Debunking Optimization Myths in Federated Learning for Medical Image Classification

Youngjoon Lee, Hyukjoon Lee, Jinu Gong, Yang Cao, Joonhyuk Kang

Abstract

Federated Learning (FL) is a collaborative learning method that enables decentralized model training while preserving data privacy. Despite its promise in medical imaging, recent FL methods are often sensitive to local factors such as optimizers and learning rates, limiting their robustness in practical deployments. In this work, we revisit vanilla FL to clarify the impact of edge device configurations, benchmarking recent FL methods on colorectal pathology and blood cell classification task. We numerically show that the choice of local optimizer and learning rate has a greater effect on performance than the specific FL method. Moreover, we find that increasing local training epochs can either enhance or impair convergence, depending on the FL method. These findings indicate that appropriate edge-specific configuration is more crucial than algorithmic complexity for achieving effective FL.

Introduction

Federated Learning (FL) is highly attractive for medical imaging because it allows hospitals to collaboratively train diagnostic models without sharing sensitive, heavily regulated patient data. To handle the inherent challenge of "non-IID" data (where different hospitals have vastly different types of patient data), researchers have recently proposed a slew of complex FL algorithms such as FedDyn and FedSAM.

However, these modern FL methods introduce complex optimization pipelines that are highly sensitive to local hyperparameters. This study addresses a critical practical question: Are these sophisticated FL algorithms actually driving performance improvements, or are we ignoring the massive impact of basic edge-device configurations? We investigate whether simply tuning local optimizers and learning rates in vanilla FL (FedAvg) is sufficient to achieve state-of-the-art results.

Methodology and Datasets

The study evaluates the original FedAvg algorithm against five recent, sophisticated methods: FedDyn, FedSAM, FedSpeed, FedSMOO, and FedGamma. The methodology focuses on systematically varying edge-device configurations in a simulated FL environment of 100 devices. Key variables include:

  • Data Heterogeneity: Both IID and highly skewed non-IID settings (using a Dirichlet distribution with $\alpha=0.1$).
  • Local Optimizers: SGD, Adam, AdamW, RMSProp, AdaGrad, and ASGD.
  • Local Learning Rates: Ranging from 0.0001 to 0.1.
  • Local Epochs: Varying training passes ($l_e \in \{1, 5, 20\}$).

Experiments were conducted on two standard medical imaging tasks: Colorectal Pathology Image Classification and Blood Cell Image Classification, with data partitioned to simulate real-world skew.

Key Findings and Results

The Impact of Local Optimizers

A primary finding is that the choice of local optimizer dictates performance much more than the choice of FL algorithm. While SGD remains the most reliable choice across all methods, modern optimizers like Adam and AdamW can severely destabilize certain algorithms. For instance, switching to Adam caused FedDyn's accuracy to plummet significantly compared to its performance with SGD, and RMSProp caused nearly all methods to fail.

Figure 3: Radar charts showing the performance of various FL algorithms across different local optimizers.

Learning Rate Sensitivity and Local Epochs

Performance is also highly sensitive to the local learning rate. Increasing the learning rate from 0.001 to 0.01 caused sharp performance drops for methods like FedSpeed and FedSMOO, while FedSAM showed more relative robustness. Furthermore, the "Local Epoch Paradox" reveals that increasing local training passes doesn't universally help; while it boosted FedAvg's accuracy, it actively degraded performance for methods like FedGamma.

Figure 6: Comparison of Top-1 Test Accuracy across different local learning rates for Colorectal Pathology and Blood Cell tasks.

Robustness of Vanilla FedAvg

Despite its simplicity, vanilla FedAvg proved remarkably robust. In IID settings, it achieved 95.02% (Colorectal) and 96.91% (Blood Cell) accuracy, trailing complex methods by very small margins. Even under high data heterogeneity, FedAvg remained competitive, often outperforming more complex methods that failed due to poor hyperparameter alignment.

Conclusion

This research suggests that the FL community should pivot from designing theoretically complex algorithms to developing methods that are resilient to variations in device-level configurations. Future work should focus on "hyperparameter-agnostic" FL that can handle the practical realities of decentralized medical networks without requiring exhaustive local tuning. While the study is limited by its simulated environment and single model backbone (ConvNeXtV2), it clearly demonstrates that appropriate edge-specific configuration is more crucial than algorithmic complexity for effective Federated Learning.

Summary updated

Abstract

Federated Learning (FL) is a collaborative learning method that enables decentralized model training while preserving data privacy. Despite its promise in medical imaging, recent FL methods are often sensitive to local factors such as optimizers and learning rates, limiting their robustness in practical deployments. In this work, we revisit vanilla FL to clarify the impact of edge device configurations, benchmarking recent FL methods on colorectal pathology and blood cell classification task. We numerically show that the choice of local optimizer and learning rate has a greater effect on performance than the specific FL method. Moreover, we find that increasing local training epochs can either enhance or impair convergence, depending on the FL method. These findings indicate that appropriate edge-specific configuration is more crucial than algorithmic complexity for achieving effective FL.
BibTeX
@misc{lee2025debunking,
  author = {Youngjoon Lee and Hyukjoon Lee and Jinu Gong and Yang Cao and Joonhyuk Kang},
  title = {{Debunking Optimization Myths in Federated Learning for Medical Image Classification}},
  year = {2025},
  eprint = {2507.19822},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.19822},
}
Figure