FedGS: Federated Gradient Scaling for Heterogeneous Medical Image Segmentation

Philip Schutte, Valentina Corbetta, Regina Beets-Tan, Wilson Silva

TL;DR

This paper introduces FedGS, a novel federated learning aggregation method that specifically addresses the challenge of segmenting small, under-represented targets in medical images. By adaptively scaling gradients based on sample difficulty, FedGS significantly improves performance on these challenging cases without compromising overall segmentation accuracy, offering a more robust solution for heterogeneous medical imaging datasets.

What Problem Does This Paper Address?

Federated Learning (FL) is a promising approach for training deep learning models on sensitive medical data, as it allows multiple institutions to collaborate without sharing raw patient information. However, a major hurdle in FL is data heterogeneity – the variations in data distributions across different clients (institutions). This heterogeneity can lead to "client drift," where local models become overly specialized to their own data, resulting in suboptimal global models that perform poorly on data from other clients, especially for under-represented cases.

In medical image segmentation, this problem is particularly acute when dealing with small lesions or tumors. These small targets are often rare in datasets, making them difficult for global models to learn effectively. Standard FL aggregation methods like Federated Averaging (FedAvg) struggle to adequately represent these minority classes, leading to missed diagnoses or inaccurate segmentations where they are most critical. Existing attempts to address heterogeneity, such as disentangled representation learning, often assume heterogeneity lies only in image "style" (e.g., acquisition protocols) and overlook variations in the "content" itself, such as the size and shape of lesions.

Methods

The core innovation of this paper is FedGS (Federated Gradient Scaling), a new aggregation strategy for FL. Unlike FedAvg, which simply averages model parameters, FedGS focuses on how to best combine the gradients from different clients. The key idea is to adaptively scale the gradients based on the estimated difficulty of the samples within a client's training batch.

Here's a breakdown of the FedGS process:

  1. Local Training: Each client trains its local model as usual, computing loss and performing backpropagation to obtain local gradients.
  2. Cumulative Gradient Tracking: Each client maintains a "cumulative gradient" (Gk) which is updated with the difference between its current and previous model parameters, scaled by a "difficulty factor" (nt). This factor is crucial.
  3. Difficulty Factor Calculation (nt): This is where FedGS shines. For each image in a training batch, a difficulty factor (δz) is calculated. This factor is primarily based on the inverse relative area of the segmented target. If the target is very small (indicated by a large inverse relative area), δz will be higher. A threshold (τ) is used to identify "small" targets. The overall batch difficulty factor (nt) is then computed by summing these δz values, giving more weight to batches with more small, difficult samples.
  4. Server Aggregation: The server collects the cumulative gradients (Gk) from all clients. It then aggregates these cumulative gradients into a global cumulative gradient (GA) using a weighted average based on the number of iterations each client performed.
  5. Global Model Update: The global model parameters are updated using this aggregated cumulative gradient.

Crucially, FedGS only modifies the gradients stored in the cumulative gradients on the client side. It does not alter the local training process itself, meaning clients can use their standard training procedures and loss functions, simplifying implementation.

Datasets

The paper evaluates FedGS on two publicly available medical image segmentation datasets:

  • PolypGen: A colonoscopy dataset from 6 different centers, focusing on polyp detection and segmentation. The authors specifically highlight its heterogeneity in patient populations and the presence of small polyps.
  • LiTS (Liver Tumor Segmentation Benchmark): A dataset of liver tumor segmentations from CT scans. This dataset is also used to evaluate performance on small targets (liver tumors).

The authors note that for PolypGen, a threshold of τ = 150 was used to classify small polyps, while for LiTS, a threshold of τ = 1000 was used for small liver tumors, reflecting the different scales of targets in each dataset.

Key Insights and Ideas

  • Difficulty-Aware Gradient Scaling: The central insight is that not all data samples are equally informative for training a global model. Small, under-represented targets are critical but difficult to learn. By scaling gradients based on this difficulty, FedGS can amplify the signal from these challenging samples.
  • Focus on Aggregation, Not Local Training: FedGS cleverly avoids altering the local training loop, which is a significant advantage. This means it can be easily integrated into existing FL frameworks without requiring clients to change their fundamental training procedures.
  • Inverse Relative Area as a Proxy for Difficulty: Using the inverse relative area of a segmentation mask as a primary indicator of difficulty is an intuitive and effective heuristic. Smaller targets inherently present a greater segmentation challenge.
  • Handling Heterogeneity Beyond Style: Unlike some prior work that focused on disentangling image style, FedGS directly addresses heterogeneity in the content of the data, specifically the size and prevalence of segmentation targets.

Key Results

The paper demonstrates that FedGS consistently outperforms FedAvg, particularly for small lesions:

  • Improved Small Lesion Segmentation (DiceS): FedGS shows significant improvements in the Dice Score for small lesions (DiceS) on both PolypGen and LiTS datasets. This is the primary contribution and a key success of the method.
  • Comparable Overall Performance (DiceL, Dice): For larger lesions (DiceL) and overall segmentation (Dice), FedGS performance is either comparable to or marginally better than FedAvg. This indicates that FedGS enhances performance on challenging cases without sacrificing general segmentation quality.
  • Dataset-Specific Improvements: The improvement in DiceS was more pronounced on the LiTS dataset, which the authors attribute to how "small" samples are defined (e.g., LiTS exclusively contains small lesions, whereas PolypGen can have large polyps with small components classified as small).
  • Marginal Runtime Overhead: FedGS introduces a small training runtime overhead (5.6% to 13.4%) compared to FedAvg, which is deemed justified by the significant improvements in segmentation performance, especially for small lesions.

Limitations and Future Work

  • Difficulty Estimation Dependency: The effectiveness of FedGS hinges on the accuracy of the difficulty estimation. While inverse relative area is a good proxy, more sophisticated difficulty metrics could potentially yield further improvements.
  • Generalizability Beyond Segmentation: The authors acknowledge that their current evaluation is limited to medical image segmentation. They suggest that FedGS holds potential for other imaging tasks and even non-imaging domains, provided that a reliable method for estimating data difficulty with respect to the downstream task can be developed.
  • Threshold Tuning: The thresholds (τ) for classifying "small" targets need to be fine-tuned for different datasets and tasks, which adds a hyperparameter to the FL setup.

Future work could explore more advanced methods for difficulty estimation, investigate the application of FedGS to other FL tasks, and potentially automate the threshold tuning process.

Summary updated

Abstract

Federated Learning (FL) in Deep Learning (DL)-automated medical image segmentation helps preserving privacy by enabling collaborative model training without sharing patient data. However, FL faces challenges with data heterogeneity among institutions, leading to suboptimal global models. Integrating Disentangled Representation Learning (DRL) in FL can enhance robustness by separating data into distinct representations. Existing DRL methods assume heterogeneity lies solely in style features, overlooking content-based variability like lesion size and shape. We propose FedGS, a novel FL aggregation method, to improve segmentation performance on small, under-represented targets while maintaining overall efficacy. FedGS demonstrates superior performance over FedAvg, particularly for small lesions, across PolypGen and LiTS datasets. The code and pre-trained checkpoints are available at the following link: https://github.com/Trustworthy-AI-UU-NKI/Federated-Learning-Disentanglement
BibTeX
@inproceedings{schutte2024fedgs,
  author = {Philip Schutte and Valentina Corbetta and Regina Beets-Tan and Wilson Silva},
  title = {{FedGS: Federated Gradient Scaling for Heterogeneous Medical Image Segmentation}},
  booktitle = {ISIC/iMIMIC/EARTH/DeCaF @ MICCAI},
  year = {2024},
  doi = {10.1007/978-3-031-77610-6\_23},
}
Figure