Federated modality-specific encoders and partially personalized fusion decoder for multimodal brain tumor segmentation

Hong Liu, Dong Wei, Qian Dai, Xian Wu, Yefeng Zheng, Liansheng Wang

Abstract

Federated learning (FL) is a promising approach for collaborative medical imaging without sharing raw data. However, clinical practice often results in "missing modalities" across different institutions, creating significant intermodal heterogeneity. We introduce FedMEPD, a novel framework that addresses this by employing modality-specific encoders, a partially personalized fusion decoder, and Localized Adaptive Calibration via Cross-Attention (LACCA). By leveraging global feature anchors, FedMEPD enables clients with incomplete data to "hallucinate" missing information. Experimental results on BraTS and HaN-Seg benchmarks demonstrate that FedMEPD significantly outperforms state-of-the-art FL methods, improving mean Dice Similarity Coefficient (mDSC) by up to 12% compared to standard FedAvg.

Introduction

In medical imaging, particularly brain tumor segmentation, a full suite of MRI sequences (T1, T1c, T2, and FLAIR) is ideal. However, real-world clinical protocols vary significantly due to cost, time, or lack of contrast agents, leading to missing modalities at different sites. This intermodal heterogeneity poses a challenge for standard Federated Learning (FL), which typically assumes uniform data types across all participants. Furthermore, institutions often require models personalized to their specific local data distributions rather than a generic global model.

Methodology

FedMEPD (Federated Modality-specific Encoders and Partially Personalized Fusion Decoder) is built upon three core components designed to handle missing data and promote effective knowledge sharing:

  1. Federated Modality-specific Encoders: Instead of a monolithic encoder, FedMEPD uses four distinct encoders tailored to each MRI sequence. Clients only train and synchronize encoders for the modalities they possess, allowing for parameter specialization.
  2. Partially Personalized Fusion Decoder: This component merges features from the encoders. It uses a consistency-based mask to decide which parameters to federate. If a filter's local updates align with global trends, it is shared; otherwise, it becomes personalized to the local client. A "patience" hyperparameter ($P$) ensures that personalization only occurs after consistent divergence, preventing overreaction to noise.
  3. Multi-Anchor Calibration (LACCA): The server generates "anchors" — representative feature prototypes for tumor regions like edema or the core. Clients use Localized Adaptive Calibration via Cross-Attention (LACCA) to align their incomplete features with these global anchors, effectively compensating for missing modality information.
Overview of the FedMEPD framework showing (a) multimodal MRI sequences and (b) the federated setup between a full-modal server and clients with missing modalities.
Detailed architecture of FedMEPD illustrating the modality-specific encoders, the partially personalized fusion decoder, and the LACCA module for anchor-based calibration.

Experimental Results

The framework was evaluated on the BraTS 2018/2020 and HaN-Seg datasets. FedMEPD consistently outperformed existing methods:

  • Accuracy Boost: On BraTS 2018, it achieved a 5% mDSC improvement over FedMSplit and a 12% improvement over standard FedAvg.
  • Server Performance: The global server model, benefiting from specialized client knowledge, reached an mDSC of 84.98%, surpassing models trained solely on local server data.
  • Robustness: The system maintained high performance even with limited server data (10%) or degraded label quality.

Discussion and Conclusion

FedMEPD effectively bridges the gap caused by missing modalities in federated environments. The use of feature anchors provides a privacy-preserving mechanism to share global knowledge without exposing raw data. While the current model assumes balanced client data, future work will focus on dynamic weight assignment for varying client sizes and further optimizing communication overhead. FedMEPD represents a significant step toward robust, personalized federated learning in heterogeneous clinical settings.

Summary updated

Abstract

Most existing federated learning (FL) methods for medical image analysis only considered intramodal heterogeneity, limiting their applicability to multimodal imaging applications. In practice, some FL participants may possess only a subset of the complete imaging modalities, posing intermodal heterogeneity as a challenge to effectively training a global model on all participants' data. Meanwhile, each participant expects a personalized model tailored to its local data characteristics in FL. This work proposes a new FL framework with federated modality-specific encoders and partially personalized multimodal fusion decoders (FedMEPD) to address the two concurrent issues. Specifically, FedMEPD employs an exclusive encoder for each modality to account for the intermodal heterogeneity. While these encoders are fully federated, the decoders are partially personalized to meet individual needs -- using the discrepancy between global and local parameter updates to dynamically determine which decoder filters are personalized. Implementation-wise, a server with full-modal data employs a fusion decoder to fuse representations from all modality-specific encoders, thus bridging the modalities to optimize the encoders via backpropagation. Moreover, multiple anchors are extracted from the fused multimodal representations and distributed to the clients in addition to the model parameters. Conversely, the clients with incomplete modalities calibrate their missing-modal representations toward the global full-modal anchors via scaled dot-product cross-attention, making up for the information loss due to absent modalities. FedMEPD is validated on the BraTS 2018 and 2020 multimodal brain tumor segmentation benchmarks. Results show that it outperforms various up-to-date methods for multimodal and personalized FL, and its novel designs are effective.
BibTeX
@article{liu2025federated,
  author = {Hong Liu and Dong Wei and Qian Dai and Xian Wu and Yefeng Zheng and Liansheng Wang},
  title = {{Federated modality-specific encoders and partially personalized fusion decoder for multimodal brain tumor segmentation}},
  journal = {Medical Image Analysis},
  year = {2025},
  doi = {10.1016/j.media.2025.103759},
}
Figure