A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentation

Mothilal Asokan, Joseph Geo Benjamin, Mohammad Yaqub, Karthik Nandakumar

TL;DR

FLAP-SAM is a new framework designed to adapt the Segment Anything Model (SAM) for 3D medical image segmentation within a Federated Learning (FL) environment. By selectively fine-tuning only a tiny fraction of the model—specifically low-rank adapters (LoRA) and the final decoder output layers—the authors achieved a 6% performance boost while slashing communication costs by 48x compared to traditional full fine-tuning.

What Problem Does This Paper Address?

The Segment Anything Model (SAM) has revolutionized 2D image segmentation, but it struggles with medical images due to "distribution shift"—medical scans look nothing like the natural images SAM was trained on. While fine-tuning helps, two major hurdles remain:

  1. Data Privacy: Medical data is often siloed in different hospitals and cannot be centralized due to privacy laws. Federated Learning (FL) allows models to train across these silos, but foundation models like SAM are massive (100M–700M parameters).
  2. Communication Bottlenecks: In FL, clients must frequently send model updates to a central server. Sending hundreds of millions of parameters over a network is slow, expensive, and often impractical for medical institutions.

Existing methods either fine-tune the whole model (too expensive) or use adapters that don't quite reach the necessary accuracy for complex 3D medical structures.

Methods

The authors introduce FLAP-SAM (Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM). Their approach relies on two main strategies:

  • Selective PEFT (Parameter-Efficient Fine-Tuning): Instead of updating the entire SAM architecture, they use LoRA (Low-Rank Adaptation) on the attention layers of the encoder and decoder. Crucially, they discovered that fine-tuning the entire decoder is actually harmful when data is scarce. Instead, they only update the final upscaling layers ($\theta_{MD-UP}$) and the mask prediction head ($\theta_{MD-HYP}$), keeping the rest of the decoder frozen.
  • FL-Optimized Aggregation: When aggregating LoRA weights from different clients, the server reconstructs the weight update matrices, averages them, and then uses Singular Value Decomposition (SVD) to compress them back into low-rank matrices before sending them back to clients. This ensures the communication remains "lean" throughout the training process.

Datasets

The researchers evaluated FLAP-SAM on three diverse 3D medical imaging benchmarks:

  • Fed-KITS2019: A 6-client federated version of the Kidney Tumor Segmentation challenge (CT scans).
  • Fed-IXI: Brain T1 MRI data collected from three different hospitals (Guys, HH, and IOP).
  • Prostate MRI: A multi-site dataset featuring T2-weighted MRI data from six different sources with varying imaging protocols.

Key Insights and Ideas

  • The "Distortion" Effect: One of the most interesting findings is that fine-tuning the entire SAM decoder on small medical datasets actually decreases performance. The authors argue that full fine-tuning distorts the powerful inherent capabilities of the foundation model. By freezing most of the decoder, FLAP-SAM retains the model's "common sense" while adapting only the necessary output parts.
  • Granular Analysis: Unlike previous works (like SAMed) that treat the decoder as a single block to be tuned, this paper performs a "surgical" analysis of which specific layers contribute most to performance.
  • Efficiency vs. Accuracy: The paper demonstrates that you don't need to trade off accuracy for efficiency. Their "partial decoder" approach actually outperformed more parameter-heavy methods.

Key Results

  • Performance Gain: On the Fed-KiTS dataset, FLAP-SAM improved the Dice score (a measure of segmentation accuracy) by ~6% compared to full fine-tuning.
  • Massive Compression: The method reduced communication overhead by ~48x compared to full fine-tuning.
  • Superior to State-of-the-Art: Compared to SAMed (a popular PEFT approach for SAM), FLAP-SAM achieved similar accuracy while using 2.8x fewer parameters and communication bandwidth.
  • Robustness: The results were consistent across different modalities (CT and MRI) and different organs (Kidney, Brain, Prostate), proving the method's versatility.

Limitations and Future Work

  • Optimization Strategies: The current work focuses on the standard FedAvg algorithm. The authors suggest that exploring more advanced federated optimization strategies (like FedProx or Scaffold) could further improve results for datasets with even more extreme distribution shifts.
  • 3D Context: While the model handles 3D data, it does so by processing stacks of 2D slices ($N=5$). Future iterations might explore more native 3D architectures while maintaining the same parameter efficiency.
  • Prompting: The study operated SAM in "fully automatic" mode. Integrating intelligent, automated prompting strategies could potentially push the accuracy even higher.

Summary updated

Abstract

Adapting foundation models for medical image analysis requires finetuning them on a considerable amount of data because of extreme distribution shifts between natural (source) data used for pretraining and medical (target) data. However, collecting task-specific medical data for such finetuning at a central location raises many privacy concerns. Although Federated learning (FL) provides an effective means for training on private decentralized data, communication costs in federating large foundation models can quickly become a significant bottleneck, impacting the solution's scalability. In this work, we address this problem of efficient communication while ensuring effective learning in FL by combining the strengths of Parameter-Efficient Fine-tuning (PEFT) with FL. Specifically, we study plug-and-play Low-Rank Adapters (LoRA) in a federated manner to adapt the Segment Anything Model (SAM) for 3D medical image segmentation. Unlike prior works that utilize LoRA and finetune the entire decoder, we critically analyze the contribution of each granular component of SAM on finetuning performance. Thus, we identify specific layers to be federated that are very efficient in terms of communication cost while producing on-par accuracy. Our experiments show that retaining the parameters of the SAM model (including most of the decoder) in their original state during adaptation is beneficial because fine-tuning on small datasets tends to distort the inherent capabilities of the underlying foundation model. On Fed-KiTS, our approach decreases communication cost (~48x) compared to full fine-tuning while increasing performance (~6% Dice score) in 3D segmentation tasks. Our approach performs similar to SAMed while achieving ~2.8x reduction in communication and parameters to be finetuned. We further validate our approach with experiments on Fed-IXI and Prostate MRI datasets.
BibTeX
@inproceedings{asokan2024federated,
  author = {Mothilal Asokan and Joseph Geo Benjamin and Mohammad Yaqub and Karthik Nandakumar},
  title = {{A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentation}},
  booktitle = {ISIC/iMIMIC/EARTH/DeCaF @ MICCAI},
  year = {2024},
  doi = {10.1007/978-3-031-77610-6\_21},
}
Figure