Which Client is Reliable?: A Reliable and Personalized Prompt-based Federated Learning for Medical Image Question Answering

He Zhu, Ren Togo, Takahiro Ogawa, Miki HaseyamaarXiv 2024 · DOI · arXiv

Introduction

Medical Artificial Intelligence (AI) currently faces a significant "double bind." On one hand, patient data is highly sensitive and subject to strict privacy and ethical regulations, making it difficult to centralize for training large-scale models. On the other hand, medical data is inherently heterogeneous; different hospitals or departments (e.g., Radiology vs. Cardiology) deal with vastly different imaging modalities and diagnostic questions. A standard "one-size-fits-all" global model often fails to capture these local nuances.

While standard Federated Learning (FL) addresses privacy by sharing model parameters instead of raw data, it typically aims for a single global model. In clinical settings, personalization is often more valuable—a model should be optimized for its specific local environment while still benefiting from the collective intelligence of other institutions. Furthermore, existing medical Visual Question Answering (VQA) models often lack the ability to quantify uncertainty, a critical requirement for clinical reliability.

Proposed Framework

To address these challenges, we propose a novel Personalized Federated Learning (pFL) framework for medical VQA that balances privacy with high performance through three key innovations.

Efficient Prompt-Tuning

Instead of updating the millions of parameters in a large Transformer model—which is computationally expensive and bandwidth-intensive—we utilize Prefix-Tuning. We insert a small set of learnable "prompts" into the Multi-Head Attention layers. During federated training, only these tiny prompts (approximately 0.01M parameters) are updated and shared, significantly reducing the communication overhead.

Uncertainty Quantification via DST

To ensure clinical reliability, we incorporate Dempster-Shafer Evidence Theory (DST). Unlike standard softmax outputs that can be overconfident, our model outputs "evidence" for each answer. This allows the system to calculate a specific uncertainty score ($u$) for every prediction, providing a measure of how "sure" the model is about its answer.

Dynamic Likelihood-weighted Uncertainty Calibration (DLUC)

This mechanism serves as the core for inter-client communication. Rather than simple weight averaging, the server uses a neural network to determine how much one client should trust another. It leverages uncertainty scores to dynamically adjust weights ($w$), ensuring that a client only integrates information from others that improves its local likelihood of being correct.

The proposed framework architecture, highlighting the shared encoder, uncertainty generation, and the DLUC mechanism for prompt updates.

Experimental Setup and Results

We evaluated our method on three major medical VQA benchmarks, partitioned to simulate different hospital departments: Slake (Lung, Abdomen, Brain, Other), VQA-RAD (Abdomen, Chest, Head), and PMC-VQA (a large-scale test with up to 500 simulated clients).

Key Findings

  • Accuracy and Efficiency: Our method achieved an average 3.5% accuracy improvement over previous state-of-the-art models while only requiring updates to 0.01% of the parameters of a standard Vision Transformer.
  • Logical Domain Relationships: Analysis of the inter-client weights showed that the DLUC mechanism learns logically sound relationships. For example, clients focusing on similar imaging features (like CT scans of the Lung and Abdomen) naturally assigned higher weights to each other's updates compared to unrelated domains like Brain imaging.
Visualization of learned weights (w) for image and question prompts across different medical domains, demonstrating how the DLUC mechanism prioritizes relevant information.

Conclusion and Future Work

Our personalized prompt-based federated learning framework demonstrates that it is possible to achieve high accuracy and clinical reliability in medical VQA without compromising data privacy. The use of uncertainty as a filter prevents irrelevant information from degrading local performance, while the prompt-tuning approach ensures extreme scalability.

Future work will focus on refining the balance between local training and inter-client communication, as the model's performance is sensitive to the aggregation rate ($\mu$). We also plan to extend this approach to even larger-scale Vision-Language Models (VLMs) to further enhance the reasoning capabilities of medical VQA systems.

Summary updated

Abstract

Conventional medical artificial intelligence (AI) models face barriers in clinical application and ethical issues owing to their inability to handle the privacy-sensitive characteristics of medical data. We present a novel personalized federated learning (pFL) method for medical visual question answering (VQA) models, addressing privacy reliability challenges in the medical domain. Our method introduces learnable prompts into a Transformer architecture to efficiently train it on diverse medical datasets without massive computational costs. Then we introduce a reliable client VQA model that incorporates Dempster-Shafer evidence theory to quantify uncertainty in predictions, enhancing the model's reliability. Furthermore, we propose a novel inter-client communication mechanism that uses maximum likelihood estimation to balance accuracy and uncertainty, fostering efficient integration of insights across clients.
BibTeX
@misc{zhu2024which,
  author = {He Zhu and Ren Togo and Takahiro Ogawa and Miki Haseyama},
  title = {{Which Client is Reliable?: A Reliable and Personalized Prompt-based Federated Learning for Medical Image Question Answering}},
  year = {2024},
  eprint = {2410.17484},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.17484},
}
Figure