Where to Begin? From Random to Foundation Model Instructed Initialization in Federated Learning for Medical Image Segmentation

Ming Li, Guang Yang

TL;DR

This paper introduces a novel initialization strategy for Federated Learning (FL) in medical imaging by using Foundation Models (like SAM) as "teachers." By distilling a large model's knowledge into a smaller student model before the federated process begins, the authors achieve significantly faster convergence and higher accuracy, particularly when dealing with the "non-IID" (heterogeneous) data common in healthcare.

What Problem Does This Paper Address?

Federated Learning is a cornerstone of privacy-preserving AI, allowing multiple hospitals to train a shared model without ever exchanging sensitive patient data. However, FL faces two major hurdles:

  1. The "Cold Start" Problem: Most FL setups begin with random initialization—essentially starting from a blank slate. This is inefficient and often leads to suboptimal performance.
  2. Data Heterogeneity (Non-IID): Medical data is rarely uniform. One clinic might have older patients (age skew), while another has a much larger volume of data (quantity skew). This lack of "Independent and Identically Distributed" (IID) data causes local models to diverge, making it hard for the global model to converge on a good solution.

While Foundation Models (FMs) like the Segment Anything Model (SAM) contain vast amounts of "pre-learned" knowledge, they are too massive and computationally expensive to be deployed directly within a standard federated network.

Methods

The authors propose FM-instructed Initialization. Instead of starting with random weights, they use a three-step process:

  1. The Teacher: They select SAM-Med2D, a version of the Segment Anything Model fine-tuned for medical images, to act as a high-knowledge "teacher."
  2. Knowledge Distillation: Because SAM-Med2D is too bulky for FL, they distill its knowledge into a compact student model called SM-lite. They use a small "proxy" dataset (data that is representative but not part of the private client data) to train the student to mimic the teacher’s outputs using KL divergence and standard segmentation loss.
  3. The Federated Start: This distilled student model—now "instructed" by the FM—serves as the starting point for the standard Federated Averaging (FedAvg) algorithm.

This approach allows the FL process to benefit from the FM's massive pre-training without the communication overhead of actually sending a giant model across the network.

Datasets

The researchers evaluated their method using the COVID-19 chest X-ray dataset, which includes 6,504 images with pixel-level lung segmentations.

  • Proxy Set: 1,504 images (used for the initial distillation).
  • Test Set: 500 images.
  • Federated Clients: 4,500 images distributed across 3 clients.
  • Simulated Challenges: To test robustness, they artificially created age skew and quantity skew among the clients to simulate real-world non-IID conditions.

Key Insights and Ideas

  • FMs as Universal Initializers: The core "aha moment" is that Foundation Models don't need to be part of the FL loop to be useful. Their knowledge can be "baked in" at the start to provide a robust feature-learning foundation.
  • Distillation is the Bridge: Distillation solves the resource constraint. It transforms "GPU-heavy" knowledge into an "FL-friendly" format.
  • Robustness to Messy Data: The FM-instructed model is significantly more resilient to data heterogeneity. Because the model starts with a strong, generalized understanding of what a "lung" looks like, it is less likely to be led astray by the specific biases or skews of an individual hospital's local dataset.

Key Results

The study compared FM-instructed initialization against random initialization and standard pre-training (training on the proxy set without an FM teacher):

  • Superior Accuracy: In the "age skew" scenario, the FM-instructed model achieved a Dice score of 85.38, significantly outperforming random initialization (74.67) and standard pre-training (83.92).
  • Closing the Gap: The performance of the FM-instructed FL model moved much closer to the "Centralized" gold standard (89.41), which assumes all data is in one place.
  • Rapid Convergence: The FM-instructed model reached high performance levels in fewer than 40 communication rounds, whereas the randomly initialized model required more than 40 rounds just to reach a much lower plateau.
  • Non-IID Resilience: While random initialization saw a sharp performance drop when moving from IID to non-IID data, the FM-instructed approach showed only marginal decreases, proving its robustness.

Limitations and Future Work

  • Proxy Data Dependency: The method relies on having a representative proxy dataset for the initial distillation phase. In some niche medical fields, even proxy data might be hard to find.
  • Task Specificity: The evaluation was limited to lung segmentation in chest X-rays. Future work could explore whether this "instruction" holds up across more complex tasks like multi-organ segmentation or 3D MRI volumes.
  • Teacher Selection: The paper specifically uses SAM-Med2D; exploring how different foundation models (or ensembles of them) affect the initialization could be a fruitful area for further research.

Summary updated

Abstract

In medical image analysis, Federated Learning (FL) stands out as a key technology that enables privacy-preserved, decentralized data processing, crucial for handling sensitive medical data. Currently, most FL models employ random initialization, which has been proven effective in various instances. However, given the unique challenges posed by non-IID (independently and identically distributed) data in FL, we propose a novel perspective: exploring the impact of using the foundation model with enormous pre-trained knowledge, such as the Segment Anything Model (SAM), as an instructive teacher for FL model initialization in medical image segmentation task. This work for the first time attempts to utilize the foundation model as an instructive teacher for initialization in FL, assessing its impact on the performance of FL models, especially in non-IID data scenarios. Our empirical evaluation on chest x-ray lung segmentation showcases that FL with foundation model instructed initialization not only achieves faster convergence but also improves performance in complex data contexts. These findings offer a new perspective for model initialization in FL.
BibTeX
@inproceedings{li2024where,
  author = {Ming Li and Guang Yang},
  title = {{Where to Begin? From Random to Foundation Model Instructed Initialization in Federated Learning for Medical Image Segmentation}},
  booktitle = {ISBI},
  year = {2024},
  url = {https://openreview.net/forum?id=lOGrwa2P10},
}

Loading related content...

Figure