A Client-Server Deep Federated Learning for Cross-Domain Surgical Image Segmentation

Ronast Subedi, Rebati Raman Gaire, Sharib Ali, Anh Nguyen, Danail Stoyanov, Binod Bhattarai

Introduction

Deep learning has revolutionized medical imaging, particularly in surgical tasks such as polyp detection and skin lesion segmentation. However, the development of robust models is often hindered by two significant challenges: strict data privacy regulations (e.g., GDPR) and "domain shift"—the performance degradation that occurs when a model trained on data from one hospital is applied to another with different imaging protocols or hardware. This paper proposes a novel federated learning (FL) framework designed to navigate these hurdles by decoupling the model architecture between a central server and local clients.

The Problem: Privacy and Domain Heterogeneity

Standard deep learning requires centralized data, which is often impossible in healthcare due to legal and ethical constraints. While Federated Learning (FL) traditionally addresses this by training models locally and aggregating updates (e.g., FedAvg), it often struggles with non-Identically and Independently Distributed (non-IID) data. In surgical imaging, differences in lighting, camera angles, and equipment noise across institutions create a significant domain gap that a single "average" global model cannot easily bridge.

Methodology: The Client-Server Architecture

The authors introduce a specialized FL framework based on an encoder-decoder structure. The core innovation lies in the distribution of these components:

1. The Source Phase

The process begins with a "source phase" where a complete model is trained on a large, well-annotated public dataset. This establishes a baseline understanding of the target anatomy.

2. The Immutable Server-Side Decoder

Following the source phase, the decoder—the part of the network responsible for generating the final segmentation mask—is uploaded to a central server and frozen. This "immutable" decoder serves as a global anchor, representing the universal knowledge of what a specific feature (like a polyp) looks like.

3. Local Client Encoders

Each participating hospital (client) maintains its own local encoder. During training, the client processes its unique images through this encoder to produce a latent representation—a compressed, abstract version of the data. This latent vector is the only information sent to the server.

4. Domain Adaptation

The server's frozen decoder processes the latent vector and calculates the loss, which is sent back to the client. The client then updates only its local encoder. This forces the encoder to learn how to map its domain-specific images into a latent space that the universal decoder can interpret, effectively "translating" local data into a global format.

Experimental Evaluation

The framework was rigorously tested across two distinct medical imaging domains:

  • Endoscopic Polyp Segmentation: Using Kvasir-SEG as the source, the model was adapted to EndoUDA, CVC-ClinicDB, and ETIS-Larib datasets. These represent a wide variety of imaging conditions and hardware.
  • Skin Lesion Segmentation: Using the ISIC dataset as the source and PH2 as the target domain.

Key Insights

  • Architectural Flexibility: The server only requires a fixed latent dimension. This allows different hospitals to use encoders of varying sizes (Small, Medium, or Large) based on their local computational resources.
  • Privacy Preservation: By transmitting abstract latent vectors rather than raw pixels, the framework significantly reduces the risk of sensitive data exposure. The authors compare this to public-key cryptography, where the encoder acts as a private key and the decoder as a public key.
  • Stability: Freezing the decoder prevents "parameter poisoning" and catastrophic forgetting, ensuring that the model remains anchored to the core task.

Results and Performance

The proposed method, particularly the variant initialized with source weights (FtEn), demonstrated superior performance compared to traditional FL methods:

  • Outperforming FedAvg: In the ETIS-Larib polyp dataset, the method achieved a 69.9 mIoU, showing greater consistency across diverse datasets where standard FedAvg and FedDG often fluctuated.
  • Superiority over Independent Training: The framework consistently outperformed models trained solely on local data (INDP), which often suffer from overfitting due to small sample sizes. This proves the value of "borrowing" the decoder's knowledge from a larger source.
  • Resource Efficiency: The system maintained high effectiveness even when clients utilized smaller, less computationally expensive encoders, making it a viable solution for smaller clinics.

Limitations and Future Directions

While promising, the authors identify several areas for further research:

  • Latent Space Security: Although safer than raw data, latent vectors may still be vulnerable to "inversion attacks." Future iterations could incorporate encryption layers.
  • Structural Constraints: The requirement for a fixed latent dimension necessitates agreement on network bottlenecks across all participating sites.
  • Communication Overhead: While efficient, the real-world bandwidth requirements for frequent latent vector transmission need further empirical study.

Conclusion

This client-server federated learning approach offers a robust solution for cross-domain surgical image segmentation. By keeping encoders local and decoders central and immutable, the framework effectively handles domain shift while maintaining high standards of data privacy and architectural flexibility.

Summary updated

Abstract

This paper presents a solution to the cross-domain adaptation problem for 2D surgical image segmentation, explicitly considering the privacy protection of distributed datasets belonging to different centers. Deep learning architectures in medical image analysis necessitate extensive training data for better generalization. However, obtaining sufficient diagnostic and surgical data is still challenging, mainly due to the inherent cost of data curation and the need of experts for data annotation. Moreover, increased privacy and legal compliance concerns can make data sharing across clinical sites or regions difficult. Another ubiquitous challenge the medical datasets face is inevitable domain shifts among the collected data at the different centers. To this end, we propose a Client-server deep federated architecture for cross-domain adaptation. A server hosts a set of immutable parameters common to both the source and target domains. The clients consist of the respective domain-specific parameters and make requests to the server while learning their parameters and inferencing. We evaluate our framework in two benchmark datasets, demonstrating applicability in computer-assisted interventions for endoscopic polyp segmentation and diagnostic skin lesion detection and analysis. Our extensive quantitative and qualitative experiments demonstrate the superiority of the proposed method compared to competitive baseline and state-of-the-art methods. We will make the code available upon the paper’s acceptance. Codes are available at: https://github.com/bhattarailab/federated-da.
BibTeX
@inproceedings{subedi2023client,
  author = {Ronast Subedi and Rebati Raman Gaire and Sharib Ali and Anh Nguyen and Danail Stoyanov and Binod Bhattarai},
  title = {{A Client-Server Deep Federated Learning for Cross-Domain Surgical Image Segmentation}},
  booktitle = {DEMI @ MICCAI},
  year = {2023},
  url = {https://openreview.net/forum?id=sJoCktqKXB},
}
Figure