FedSaaS: Class-Consistency Federated Semantic Segmentation via Global Prototype Supervision and Local Adversarial Harmonization

Xiaoyang Yu, Xiaoming Wu, Xin Wang, Dongrun Li, Ming Yang, Peng ChengIJCAI 2025 · DOI

TL;DR

This paper introduces FedSaaS, a novel federated learning framework for semantic segmentation that tackles the challenge of inconsistent class representations across different clients due to domain shift. By leveraging class exemplars for global prototype supervision and local adversarial harmonization, FedSaaS significantly improves segmentation accuracy and ensures consistent performance across diverse datasets.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a category label to each pixel in an image, is crucial for applications like autonomous driving and medical imaging. While deep learning has advanced this field, training models requires vast amounts of labeled data, raising privacy concerns. Federated learning (FL) offers a solution by enabling collaborative training without sharing raw data. However, a major hurdle in federated semantic segmentation is data heterogeneity across clients, often manifesting as domain shift (e.g., different lighting, weather, or camera angles) and label skew. This heterogeneity leads to inconsistent representations of the same object class in the semantic space, making it difficult for models to generalize and perform accurately across all clients. Existing FL methods often focus on global generalization or client-specific personalization but overlook the fine-grained alignment of class representations between local (client-specific) and global (server-aggregated) models, leading to semantic mismatches.

Methods

FedSaaS introduces a two-pronged approach to achieve class consistency:

  • Global Prototype Supervision:
    • Class Exemplars: The framework utilizes "class exemplars" – representative samples of each class derived from client data – as a key component.
  • Server-side Training: On the server, these exemplars are used to train a global model and generate class prototypes. These prototypes capture the essence of each class's representation.
  • Supervision: The server-side class prototypes are then sent back to clients to supervise their global branches, ensuring that client-side global representations align with the server's understanding. This is achieved through a weighted ensemble distillation process and inter-client contrastive learning on the server.
  • Local Adversarial Harmonization:
    • Client-side Balancing: On each client, the model has both a local and a global branch. To harmonize their contributions, an adversarial mechanism is employed.
  • Discriminator: A discriminator is trained to distinguish between the outputs of the local and global branches.
  • Branches' Objective: The local and global branches are then trained to "fool" the discriminator, forcing them to produce more aligned and consistent outputs.
  • Multilevel Contrastive Losses: Both client and server sides utilize multilevel contrastive losses based on class exemplars to further enforce consistency within the semantic space.

Datasets

The paper evaluates FedSaaS on driving scene segmentation datasets, specifically addressing two heterogeneity scenarios:

  • Slight Heterogeneity: Cityscapes dataset, known for its relatively minor domain shift.
  • Severe Heterogeneity: A combination of five datasets: Cityscapes, Mapillary Vistas, BDD100K, GTA5, and Synthia. This combination introduces significant domain and label shifts, providing a more rigorous test of the model's generalization capabilities.

The paper also uses Cityscapes as an "unseen domain" for testing generalization.

Key Insights and Ideas

  • Class Exemplars as a Unifying Concept: The core innovation is using class exemplars not just for representation but as a bridge between local and global learning. They provide a concrete basis for aligning disparate client data.
  • Prototype Supervision for Global Alignment: The idea of using server-generated "class prototypes" to guide client-side global branches is a powerful way to enforce a shared understanding of classes, combating domain shift.
  • Adversarial Harmonization for Local-Global Synergy: The adversarial approach on the client side is crucial for balancing the specific local features with the generalized global features, preventing one from dominating the other.
  • Two-Level Consistency is Key: The paper emphasizes that achieving consistency at both the local and global levels within the semantic space is essential for robust federated semantic segmentation.

Key Results

  • Superior Performance: FedSaaS significantly outperforms state-of-the-art federated semantic segmentation methods across both slight and severe heterogeneity scenarios.
  • Significant Improvements: In the severe heterogeneity case, FedSaaS achieved improvements of 6.68% in Accuracy and 4.48% in mIoU compared to other methods.
  • Effective Module Contributions: Ablation studies show that each proposed module (prototype supervision, contrastive loss, adversarial harmonization) contributes to performance gains, with the combined approach yielding the best results.
  • Communication Efficiency: FedSaaS demonstrates improved performance with fewer communication rounds, and even with reduced upload ratios of class exemplars (around 50-75%), it maintains high accuracy, suggesting practical applicability.
  • Qualitative Improvements: Visualizations (Figure 4 and 6) show that FedSaaS produces more accurate segmentations and better feature attention compared to baseline methods, especially in challenging scenarios.

Limitations and Future Work

  • Communication Overhead: While efficient, uploading class exemplars does introduce some communication overhead. The paper investigates this by showing that partial uploads are still effective, but further optimization might be explored.
  • Initial Instability: The adversarial training can lead to initial instability and transient loss fluctuations, particularly under severe heterogeneity. The authors mitigate this with strategies like gradual weight increase for the adversarial loss and gradient clipping, but it remains an area to consider.
  • Future Directions: The paper implicitly suggests future work in exploring more sophisticated methods for exemplar selection, further optimizing the adversarial harmonization, and potentially extending the framework to other federated learning tasks beyond semantic segmentation.

Summary updated

BibTeX
@inproceedings{yu2025fedsaas,
  author = {Xiaoyang Yu and Xiaoming Wu and Xin Wang and Dongrun Li and Ming Yang and Peng Cheng},
  title = {{FedSaaS: Class-Consistency Federated Semantic Segmentation via Global Prototype Supervision and Local Adversarial Harmonization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/770},
}
Figure