Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classification

Naif Alkhunaizi, Faris Almalik, Rouqaiah Al-Refai, Muzammal Naseer, Karthik Nandakumar

Abstract

With the advent of large pre-trained transformer models, fine-tuning these models for various downstream tasks is a critical problem. Paucity of training data, the existence of data silos, and stringent privacy constraints exacerbate this fine-tuning problem in the medical imaging domain, creating a strong need for algorithms that enable collaborative fine-tuning of pre-trained models. Moreover, the large size of these models necessitates the use of parameter-efficient fine-tuning (PEFT) to reduce the communication burden in federated learning. In this work, we systematically investigate various federated PEFT strategies for adapting a Vision Transformer (ViT) model (pre-trained on a large natural image dataset) for medical image classification. Apart from evaluating known PEFT techniques, we introduce new federated variants of PEFT algorithms such as visual prompt tuning (VPT), low-rank decomposition of visual prompts, stochastic block attention fine-tuning, and hybrid PEFT methods like low-rank adaptation (LoRA)+VPT. Moreover, we perform a thorough empirical analysis to identify the optimal PEFT method for the federated setting and understand the impact of data distribution on federated PEFT, especially for out-of-domain (OOD) and non-IID data. The key insight of this study is that while most federated PEFT methods work well for in-domain transfer, there is a substantial accuracy vs. efficiency trade-off when dealing with OOD and non-IID scenarios, which is commonly the case in medical imaging. Specifically, every order of magnitude reduction in fine-tuned/exchanged parameters can lead to a 4% drop in accuracy. Thus, the initial model choice is crucial for federated PEFT. It is preferable to use medical foundation models learned from in-domain medical image data (if available) rather than general vision models.

Introduction

The deployment of deep learning in healthcare is often hindered by the "data silo" problem. While large-scale datasets are essential for training robust models, medical data is frequently fragmented across institutions and protected by stringent privacy regulations such as HIPAA and GDPR. Federated Learning (FL) has emerged as a promising paradigm to address these challenges, enabling collaborative model training without the need for centralized data sharing.

However, the transition to state-of-the-art architectures like Vision Transformers (ViTs) introduces a significant hurdle: communication overhead. ViTs often contain hundreds of millions of parameters. In a federated setting, transmitting these full models between clients and a central server in every training round is prohibitively expensive. Parameter-Efficient Fine-Tuning (PEFT) offers a solution by updating only a small subset of parameters. While PEFT has shown remarkable success in natural image domains, its efficacy in the specialized and often "out-of-domain" (OOD) context of medical imaging remains under-explored.

Federated PEFT Methodology

This study systematically benchmarks several PEFT strategies within a federated framework. The goal is to adapt a ViT-B/16 model, pre-trained on ImageNet-21k, to medical image classification tasks.

Overview of the federated PEFT framework for Vision Transformers, illustrating various fine-tuning strategies like LoRA and VPT within a federated learning architecture.

The researchers evaluated several existing and novel techniques:

  • LoRA (Low-Rank Adaptation): Injects trainable low-rank matrices into the transformer layers, significantly reducing the number of parameters to be updated.
  • VPT (Visual Prompt Tuning): Adds a small set of learnable "prompt" tokens to the input sequence of each transformer layer.
  • SBA (Stochastic Block Attention): A novel method where the server randomly selects a single attention block for clients to update in each round, minimizing per-round communication.
  • DVPT (Decomposed Visual Prompts): Utilizes Singular Value Decomposition (SVD) to compress visual prompts, further enhancing parameter efficiency.
  • Hybrid Approaches: Combinations such as LoRA + VPT were tested to explore synergistic effects between weight-based and prompt-based tuning.

The evaluation spanned two domains: Medical (OOD) using the Fed-ISIC2019 and HAM10000 skin lesion datasets, and Natural (In-Domain) using Caltech101 and Flowers102 as baselines.

The Accuracy-Efficiency Trade-off

A primary finding of this research is the stark contrast between in-domain and out-of-domain performance. In natural image tasks (In-Domain), PEFT methods can reduce trainable parameters by several orders of magnitude with negligible impact on accuracy—a phenomenon referred to as the "In-Domain Illusion."

Accuracy vs. number of trainable parameters for in-domain (Caltech101) and out-of-domain (Fed-ISIC2019) datasets, highlighting the significant performance drop for OOD medical data as parameters are reduced.

However, for medical imaging (OOD), a significant "accuracy tax" is observed. The empirical data reveals that for every order of magnitude reduction in the number of exchanged parameters, there is approximately a 4% drop in balanced accuracy. This suggests that while PEFT is viable, the trade-off between communication efficiency and model performance is much more severe in specialized domains.

Key Results and Insights

  • Performance Leaders: All Blocks Attention (ABA) achieved the highest accuracy on medical datasets (e.g., ~81% on HAM10000), but at the cost of high communication (28M parameters).
  • Efficiency Champions: LoRA proved highly efficient, utilizing only 0.18% of the full model's parameters, though it suffered a ~5% accuracy drop on the challenging Fed-ISIC2019 dataset compared to full fine-tuning.
  • Robustness of Visual Prompts: Visual prompts (VPT) were found to be more effective than text-based prompts (PromptFL) for medical tasks, likely because visual features in clinical imagery are better captured by the model's internal visual representations than by natural language descriptions.
  • Importance of Initialization: The study emphasizes that the starting weights are critical. PEFT methods are most effective when the pre-trained model is already close to the target domain. Consequently, using "Medical Foundation Models" pre-trained on clinical data is recommended over general-purpose models.

Conclusion

This work provides a comprehensive analysis of federated PEFT for medical ViTs, uncovering a critical accuracy-efficiency trade-off in OOD scenarios. While PEFT significantly mitigates communication bottlenecks, the "4% penalty" per order of magnitude parameter reduction must be carefully considered in clinical applications. Future research should focus on developing medical-specific PEFT modules and exploring these strategies in larger-scale, cross-device federated settings to further bridge the gap between efficiency and diagnostic accuracy.

Summary updated

Abstract

With the advent of large pre-trained transformer models, fine-tuning these models for various downstream tasks is a critical problem. Paucity of training data, the existence of data silos, and stringent privacy constraints exacerbate this fine-tuning problem in the medical imaging domain, creating a strong need for algorithms that enable collaborative fine-tuning of pre-trained models. Moreover, the large size of these models necessitates the use of parameter-efficient fine-tuning (PEFT) to reduce the communication burden in federated learning. In this work, we systematically investigate various federated PEFT strategies for adapting a Vision Transformer (ViT) model (pre-trained on a large natural image dataset) for medical image classification. Apart from evaluating known PEFT techniques, we introduce new federated variants of PEFT algorithms such as visual prompt tuning (VPT), low-rank decomposition of visual prompts, stochastic block attention fine-tuning, and hybrid PEFT methods like low-rank adaptation (LoRA)+VPT. Moreover, we perform a thorough empirical analysis to identify the optimal PEFT method for the federated setting and understand the impact of data distribution on federated PEFT, especially for out-of-domain (OOD) and non-IID data. The key insight of this study is that while most federated PEFT methods work well for in-domain transfer, there is a substantial accuracy vs. efficiency trade-off when dealing with OOD and non-IID scenarios, which is commonly the case in medical imaging. Specifically, every order of magnitude reduction in fine-tuned/exchanged parameters can lead to a 4% drop in accuracy. Thus, the initial model choice is crucial for federated PEFT. It is preferable to use medical foundation models learned from in-domain medical image data (if available) rather than general vision models.
BibTeX
@inproceedings{alkhunaizi2024probing,
  author = {Naif Alkhunaizi and Faris Almalik and Rouqaiah Al-Refai and Muzammal Naseer and Karthik Nandakumar},
  title = {{Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classification}},
  booktitle = {ISIC/iMIMIC/EARTH/DeCaF @ MICCAI},
  year = {2024},
  doi = {10.1007/978-3-031-77610-6\_22},
}
Figure