X-FLoRA: Cross-modal Federated Learning with Modality-expert LoRA for Medical VQA

Min Hyuk Kim, Changheon Kim, Seok Bong YooEMNLP 2025

TL;DR

This paper introduces X-FLoRA, a novel federated learning framework for medical Visual Question Answering (VQA) that tackles the challenge of cross-modal data heterogeneity. By combining federated asymmetric translation with modality-expert LoRA fine-tuning, X-FLoRA enables accurate VQA even when clients only possess data from a single imaging modality (like MRI or CT), without requiring data sharing.

What Problem Does This Paper Address?

Medical VQA holds immense promise for computer-aided diagnosis, but its widespread adoption is hindered by data privacy concerns and the inherent difficulties in training models on diverse medical imaging modalities. Federated Learning (FL) offers a privacy-preserving solution by training models collaboratively across institutions without sharing raw patient data. However, existing FL approaches struggle in "cross-modal" scenarios where different clients have access to different imaging modalities (e.g., some have MRI, others have CT) and lack paired data across these modalities. This heterogeneity, stemming from different imaging physics and semantic focuses in reports, makes it difficult to build a unified VQA model. Traditional domain adaptation methods alone are insufficient, and simply aggregating models from different modalities leads to performance degradation.

Methods

X-FLoRA addresses this challenge through a two-phase approach:

  1. Federated Asymmetric Translation:
    • Goal: Synthesize images from one modality to another without sharing raw data.
  2. Mechanism: Each client trains a backward translation model (e.g., CT-to-MRI) using their own data and associated EMR reports. The forward translation model is kept frozen. This process leverages clinical semantics from textual reports to enrich the synthetic images.
  3. Novelty: Asymmetric training (only backward model updated) and text-driven attention for richer synthetic data.
  4. Aggregation: Backward translation weights are aggregated at the server using a discriminator quality score-based approach, prioritizing clients with better translation models.
  5. Federated VQA Fine-tuning with Modality-expert LoRA:
    • Goal: Enhance modality-specific representations for VQA.
  6. Mechanism: Clients use the synthesized images from Phase 1 to fine-tune modality-expert Low-Rank Adaptation (LoRA) modules. Separate LoRA modules are trained for MRI, CT, and text encoders, allowing them to learn discriminative features specific to each modality.
  7. Novelty: Modality-expert LoRA, a lightweight adaptation technique that fine-tunes specific layers (e.g., attention and feedforward blocks) of the encoders, preserving the general capabilities of the backbone VLM.
  8. Aggregation: LoRA weights are aggregated in a modality-specific manner (separately for MRI, CT, and text) using an expert-aware weighting scheme that balances contributions from real and synthetic data.

Datasets

  • LLaVA-Med dataset: A large-scale dataset designed for multimodal learning across institutions, featuring instruction-following data derived from PubMed Central, including ~10K samples across CT and MRI modalities.
  • VQA-RAD dataset: Contains 3,515 clinician-authored QA pairs and 315 radiology images with GPT-4 generated reports.

Key Insights and Ideas

  • Text is Crucial for Cross-Modal Translation: The paper demonstrates that incorporating textual reports significantly improves the quality of synthesized medical images, preserving clinical insights that might be lost in image-only translation.
  • Modality-Specific Expertise Matters: Treating each modality (MRI, CT, text) with specialized fine-tuning modules (LoRA) is more effective than a one-size-fits-all approach, allowing the model to capture unique characteristics.
  • Lightweight Adaptation is Key for FL: LoRA's low-rank adaptation significantly reduces trainable parameters and computational overhead, making it ideal for resource-constrained federated learning environments.
  • Federated Asymmetric Translation is a Powerful Data Augmentation: Synthesizing data in a privacy-preserving manner effectively augments the limited, single-modality data available at each client, leading to better VQA performance.

Key Results

  • State-of-the-Art Performance: X-FLoRA outperforms existing federated learning methods on both LLaVA-Med and VQA-RAD datasets across multiple metrics (BLEU, METEOR, ROUGE, CIDEr, and Accuracy).
  • Quantitative Improvements: For example, on LLaVA-Med, X-FLoRA achieved a CIDEr score of 0.5430, surpassing other methods like FedTGP (0.5237) and IOS (0.5190). On VQA-RAD, X-FLoRA achieved 81.10% accuracy, outperforming FedTGP (78.50%) and IOS (78.37%).
  • Computational Efficiency: X-FLoRA requires significantly fewer trainable parameters (58M) compared to other methods (e.g., 13G) and converges faster.
  • Qualitative Improvements: Figures in the paper show X-FLoRA generating more accurate and clinically relevant answers, often matching GPT-4's ground truth, even in complex cases.

Limitations and Future Work

  • Clinical Interpretability: The authors acknowledge that the clinical interpretability and reliability of generated responses have not been directly assessed by expert review. Future work should include qualitative evaluations by domain experts (radiologists, physicians) to understand how model outputs are perceived and trusted.
  • Broader Modality Support: While X-FLoRA is extensible, current experiments focus on MRI and CT. Future work aims to extend the framework to other modalities like ultrasound, PET, and digital pathology, which presents challenges in mapping diverse representations to a common space. Advanced feature alignment techniques are proposed to address this.
  • Retrieval-Augmented Generation (RAG): Integrating RAG into FL for medical VQA is challenging due to privacy constraints on data sharing and the need for a centralized corpus. Future work will explore privacy-preserving retrieval methods.

Summary updated

BibTeX
@inproceedings{kim2025x,
  author = {Min Hyuk Kim and Changheon Kim and Seok Bong Yoo},
  title = {{X-FLoRA: Cross-modal Federated Learning with Modality-expert LoRA for Medical VQA}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure