A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentation

Mothilal Asokan, Joseph Geo Benjamin, Mohammad Yaqub, Karthik Nandakumar

Abstract

Adapting the Segment Anything Model (SAM) for 3D medical imaging presents significant challenges due to distribution shifts, data privacy concerns, and the massive communication overhead inherent in Federated Learning (FL). This paper introduces FLAP-SAM, a framework designed for parameter-efficient fine-tuning (PEFT) that selectively updates a fraction of the model's parameters. By focusing on low-rank adapters (LoRA) and specific decoder output layers, FLAP-SAM achieves a 6% performance improvement in segmentation accuracy while reducing communication costs by 48x compared to full fine-tuning.

Introduction

The Segment Anything Model (SAM) has set new benchmarks in 2D natural image segmentation. However, its application to the medical domain is hindered by a pronounced "distribution shift," as medical scans (CT, MRI) differ fundamentally from the natural images in SAM's training set. Furthermore, medical data is typically siloed across institutions due to strict privacy regulations, making centralized training impossible.

Federated Learning (FL) offers a solution by allowing models to train across decentralized clients. Yet, the sheer size of foundation models like SAM (ranging from 100M to 700M parameters) creates a communication bottleneck. Sending full model updates between clients and a central server is often impractical for medical institutions with limited bandwidth. FLAP-SAM addresses these hurdles by optimizing both the fine-tuning process and the communication protocol.

Methodology: The FLAP-SAM Framework

FLAP-SAM (Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM) utilizes two core strategies to balance accuracy and efficiency:

1. Selective Parameter-Efficient Fine-Tuning (PEFT)

Rather than updating the entire SAM architecture, FLAP-SAM employs Low-Rank Adaptation (LoRA) on the attention layers of both the encoder and decoder. A critical finding of this research is that fine-tuning the entire decoder can be counterproductive when training data is scarce. The authors observed a "distortion effect" where full fine-tuning degrades the pre-trained model's robust feature representations.

To mitigate this, FLAP-SAM performs a "surgical" update, freezing most of the decoder and only fine-tuning:

  • The final upscaling layers ($\theta_{MD-UP}$)
  • The mask prediction head ($\theta_{MD-HYP}$)

2. FL-Optimized Aggregation

To maintain lean communication, the framework optimizes how weights are aggregated. When the central server receives LoRA updates, it reconstructs the weight matrices, averages them, and applies Singular Value Decomposition (SVD). This compresses the updates back into low-rank matrices before they are redistributed to clients, ensuring that the communication remains efficient throughout the training lifecycle.

Experimental Setup and Datasets

The effectiveness of FLAP-SAM was evaluated across three diverse 3D medical imaging benchmarks, simulating real-world federated environments:

  • Fed-KITS2019: A 6-client federated version of the Kidney Tumor Segmentation challenge using CT scans.
  • Fed-IXI: Brain T1 MRI data sourced from three distinct hospitals (Guys, HH, and IOP).
  • Prostate MRI: A multi-site dataset featuring T2-weighted MRI data from six different sources with varying imaging protocols.

Key Insights and Results

Overcoming the Distortion Effect

The study reveals that for complex 3D structures, "less is more." By freezing the majority of the decoder, FLAP-SAM retains the foundation model's "common sense" while adapting only the necessary output components for specific medical tasks. This partial decoder approach actually outperformed more parameter-heavy methods.

Performance and Efficiency Metrics

  • Accuracy: On the Fed-KiTS dataset, FLAP-SAM improved the Dice score by approximately 6% compared to traditional full fine-tuning.
  • Communication Efficiency: The method achieved a 48x reduction in communication overhead compared to full fine-tuning.
  • Benchmarking: When compared to SAMed—a leading PEFT approach—FLAP-SAM delivered comparable accuracy while utilizing 2.8x fewer parameters and significantly less bandwidth.
  • Generalization: The framework demonstrated robustness across different modalities (CT and MRI) and various anatomical structures (kidney, brain, and prostate).

Conclusion and Future Work

FLAP-SAM demonstrates that high-performance 3D medical segmentation can be achieved within a federated framework without the prohibitive costs of full model fine-tuning. By strategically selecting layers for adaptation and utilizing SVD-based compression, the framework provides a scalable path for deploying foundation models in sensitive medical environments.

Future research may explore integrating advanced federated optimization strategies like FedProx to handle extreme data heterogeneity. Additionally, while the current model processes 3D data as stacks of 2D slices, future iterations could investigate native 3D architectures and automated prompting strategies to further enhance segmentation precision.

Summary updated

BibTeX
@inproceedings{asokan2024federated,
  author = {Mothilal Asokan and Joseph Geo Benjamin and Mohammad Yaqub and Karthik Nandakumar},
  title = {{A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentation}},
  booktitle = {ISIC/iMIMIC/EARTH/DeCaF @ MICCAI},
  year = {2024},
  doi = {10.1007/978-3-031-77610-6\_21},
}
Figure