Abstract
Medical Visual Question Answering (VQA) faces significant challenges due to data heterogeneity across different medical institutions and the high communication costs associated with large-scale Transformer models. This paper presents a novel prompt-based personalized federated learning (pFL) framework designed to address these issues. By sharing small, learnable "prompts" instead of full model weights, the proposed method achieves high accuracy across diverse medical datasets while reducing communication overhead to just 0.05% of traditional federated learning methods. The framework incorporates a reliability parameter and a distance loss function to ensure robust knowledge transfer and maintain local task relevance.
Introduction
Medical VQA systems, which answer natural language questions about medical images, are essential for clinical decision support. However, two primary hurdles impede their deployment: data heterogeneity and privacy concerns. Medical data is often siloed in different institutions, and images of different organs (e.g., brain MRIs vs. lung CTs) exhibit vastly different feature distributions. While Federated Learning (FL) allows for collaborative training without moving sensitive patient data, traditional FL methods struggle with the massive parameter counts of modern Transformer-based models, leading to prohibitive communication costs.
This research introduces a personalized federated learning approach that utilizes prompt-based learning to achieve efficiency and high performance across heterogeneous medical domains.
Methodology
The core of the proposed framework is the shift from full-weight sharing to the exchange of succinct "prompts" through Prompt-based Residual Attention (pRA) blocks.
Prompt-based pFL
Instead of updating and sharing the millions of parameters within a Transformer encoder (such as CLIP), each client trains a small set of learnable prompt vectors. These prompts are injected into the multi-head self-attention (MHA) layers of the model. During the federated process, only these small prompt vectors are exchanged between clients and the server, drastically reducing bandwidth requirements.
Reliability and Similarity
To mitigate the risk of "negative transfer"—where a client with poor data or a different task degrades the global model—the authors introduced a reliability parameter ($\eta$). This weight is determined by:
- Local Accuracy: The performance of the client on its own data.
- Cosine Similarity: The similarity between a client's prompts and those of others.
This mechanism ensures that clients prioritize knowledge from others working on similar anatomical regions or demonstrating high performance.
Distance Loss ($L_d$)
A specific distance loss function is employed to ensure that local prompts do not drift too far from the shared global prompts. This maintains a balance between the specialized local knowledge and the generalized knowledge gained from the federated network.
Experimental Setup
The framework was evaluated using two major medical VQA benchmarks, partitioned to simulate multi-institutional heterogeneity:
- Slake: Divided into four clients based on anatomy: Lung, Abdomen, Brain, and Others.
- VQA-RAD: Divided into three clients: Chest, Abdomen, and Brain.
These datasets include closed-type question-answer pairs paired with various imaging modalities like CT and MRI.
Results and Discussion
Performance and Efficiency
The proposed method (PM) demonstrated superior performance compared to state-of-the-art medical VQA models, including PMCLIP and M2I2. For instance, on the Slake "Lung" client, the method achieved 84.40% accuracy, a significant improvement over the VGG+SAN baseline of 75.89%.
Crucially, the method achieved these results with extreme efficiency. By updating only the prompts, the system shared only 0.05% of the parameters required by traditional pFL methods that exchange full model weights. This makes the framework highly suitable for real-world medical environments with limited network bandwidth.
Personalization and Ablation
t-SNE visualizations of the learned prompts revealed that they naturally clustered by organ type, confirming that the prompts effectively act as "task signatures" for specific anatomical regions. Ablation studies (AS1-AS4) further confirmed that both the federated communication component and the prompt-based architecture were vital; removing either led to a measurable decline in accuracy.
Conclusion
This paper demonstrates that prompt-based personalized federated learning is a highly effective strategy for medical VQA. By focusing on small, learnable prompts and incorporating reliability-weighted sharing, the framework overcomes the dual challenges of data heterogeneity and communication overhead. Future work will explore the scalability of this approach to hundreds of clients and the potential for adaptive prompt lengths to further optimize performance across diverse medical tasks.
Summary updated