Introduction
Medical Artificial Intelligence (AI) currently faces a significant "double bind." On one hand, patient data is highly sensitive and subject to strict privacy and ethical regulations, making it difficult to centralize for training large-scale models. On the other hand, medical data is inherently heterogeneous; different hospitals or departments (e.g., Radiology vs. Cardiology) deal with vastly different imaging modalities and diagnostic questions. A standard "one-size-fits-all" global model often fails to capture these local nuances.
While standard Federated Learning (FL) addresses privacy by sharing model parameters instead of raw data, it typically aims for a single global model. In clinical settings, personalization is often more valuable—a model should be optimized for its specific local environment while still benefiting from the collective intelligence of other institutions. Furthermore, existing medical Visual Question Answering (VQA) models often lack the ability to quantify uncertainty, a critical requirement for clinical reliability.
Proposed Framework
To address these challenges, we propose a novel Personalized Federated Learning (pFL) framework for medical VQA that balances privacy with high performance through three key innovations.
Efficient Prompt-Tuning
Instead of updating the millions of parameters in a large Transformer model—which is computationally expensive and bandwidth-intensive—we utilize Prefix-Tuning. We insert a small set of learnable "prompts" into the Multi-Head Attention layers. During federated training, only these tiny prompts (approximately 0.01M parameters) are updated and shared, significantly reducing the communication overhead.
Uncertainty Quantification via DST
To ensure clinical reliability, we incorporate Dempster-Shafer Evidence Theory (DST). Unlike standard softmax outputs that can be overconfident, our model outputs "evidence" for each answer. This allows the system to calculate a specific uncertainty score ($u$) for every prediction, providing a measure of how "sure" the model is about its answer.
Dynamic Likelihood-weighted Uncertainty Calibration (DLUC)
This mechanism serves as the core for inter-client communication. Rather than simple weight averaging, the server uses a neural network to determine how much one client should trust another. It leverages uncertainty scores to dynamically adjust weights ($w$), ensuring that a client only integrates information from others that improves its local likelihood of being correct.
Experimental Setup and Results
We evaluated our method on three major medical VQA benchmarks, partitioned to simulate different hospital departments: Slake (Lung, Abdomen, Brain, Other), VQA-RAD (Abdomen, Chest, Head), and PMC-VQA (a large-scale test with up to 500 simulated clients).
Key Findings
- Accuracy and Efficiency: Our method achieved an average 3.5% accuracy improvement over previous state-of-the-art models while only requiring updates to 0.01% of the parameters of a standard Vision Transformer.
- Logical Domain Relationships: Analysis of the inter-client weights showed that the DLUC mechanism learns logically sound relationships. For example, clients focusing on similar imaging features (like CT scans of the Lung and Abdomen) naturally assigned higher weights to each other's updates compared to unrelated domains like Brain imaging.
Conclusion and Future Work
Our personalized prompt-based federated learning framework demonstrates that it is possible to achieve high accuracy and clinical reliability in medical VQA without compromising data privacy. The use of uncertainty as a filter prevents irrelevant information from degrading local performance, while the prompt-tuning approach ensures extreme scalability.
Future work will focus on refining the balance between local training and inter-client communication, as the model's performance is sensitive to the aggregation rate ($\mu$). We also plan to extend this approach to even larger-scale Vision-Language Models (VLMs) to further enhance the reasoning capabilities of medical VQA systems.
Summary updated