FedSeg: Class-Heterogeneous Federated Learning for Semantic Segmentation

Jiaxu Miao, Zongxin Yang, Leilei Fan, Yi Yang

Introduction: The Challenge of Class Heterogeneity

Federated Learning (FL) enables collaborative model training across distributed parties without sharing raw data. While successful in image classification, semantic segmentation introduces a unique obstacle: Class Heterogeneity. In real-world scenarios, different clients (e.g., autonomous vehicles in different cities) may only encounter a subset of object classes. For instance, Client A might see "motorbikes" while Client B sees "pedestrians."

Conceptual overview of class heterogeneity and the resulting optimization divergence in pixel embedding space.

In standard FL, Client A would label pedestrians as "background," forcing its local model to suppress these features. When these conflicting updates are aggregated, the global model suffers from severe "client drift" and poor convergence. Existing weight-level regularization methods are often too coarse to resolve these fine-grained, dense prediction conflicts.

Methodology: The FedSeg Framework

To mitigate these issues, FedSeg introduces two core innovations to the Federated Averaging (FedAvg) pipeline:

1. Modified Cross-Entropy Loss ($L_{backce}$)

Standard cross-entropy ignores background pixels, which in this context are actually "unseen" foreground classes. FedSeg aggregates the probabilities of all classes not present in a client's local dataset and treats the background as a mixture of these missing classes. This provides a negative gradient signal that prevents the local model from incorrectly suppressing features of classes it hasn't seen yet.

2. Local-to-Global Pixel Contrastive Learning ($L_{con}$)

To maintain alignment with the global semantic space, FedSeg uses contrastive learning. Local pixel embeddings are pulled toward the global representation of the same class and pushed away from others. To ensure computational efficiency, the authors use a Pixel-to-Region approach: local pixels are compared against compact "region embeddings" (class anchors) derived from the global model, rather than a dense pixel-to-pixel comparison.

The FedSeg architecture, illustrating the local model update process with modified loss functions and global model interaction.

Experimental Evaluation

FedSeg was evaluated on four major benchmarks: Cityscapes, CamVID, PascalVOC, and ADE20k. These datasets were partitioned to simulate extreme class heterogeneity, where clients might only observe 1 or 2 classes.

Key Results

  • Significant Accuracy Gains: On the Cityscapes dataset under non-IID conditions, FedSeg achieved a 50.24% mIoU, a massive leap from the 10.40% achieved by standard FedAvg.
  • Superior Convergence: FedSeg consistently outperformed state-of-the-art FL algorithms like FedProx, FedDyn, and MOON by 4-5% mIoU across various settings.
  • Efficiency: The framework demonstrates faster convergence, often reaching target accuracy in half the communication rounds required by competing methods.

Insights and Discussion

The primary insight of FedSeg is that the "background" in heterogeneous FL is not empty but contains hidden foreground classes. Addressing this at the loss-function level is more effective than architectural changes. Furthermore, using the global model as a "semantic anchor" provides a reliable teacher for local clients.

Limitations

While effective, FedSeg's contrastive component is sensitive to the quality of pseudo-labels generated by the global model. Additionally, while it solves class heterogeneity, future work is needed to address domain heterogeneity (e.g., varying lighting or weather conditions across clients) and open-world scenarios where the class set is not static.

Summary updated

Abstract

Federated Learning (FL) is a distributed learning paradigm that collaboratively learns a global model by multiple clients with data privacy-preserving. Although many FL algorithms have been proposed for classification tasks, few works focus on more challenging semantic segmentation tasks, especially in the class-heterogeneous FL situation. Compared with classification, the issues from heterogeneous FL for semantic segmentation are more severe: (1) Due to the non-IID distribution, different clients may contain inconsistent foreground-background classes, resulting in divergent local updates. (2) Class-heterogeneity for complex dense prediction tasks makes the local optimum of clients farther from the global optimum. In this work, we propose FedSeg, a basic federated learning approach for class-heterogeneous semantic segmentation. We first propose a simple but strong modified cross-entropy loss to correct the local optimization and address the foreground-background inconsistency problem. Based on it, we introduce pixel-level contrastive learning to enforce local pixel embeddings belonging to the global semantic space. Extensive experiments on four semantic segmentation benchmarks (Cityscapes, CamVID, PascalVOC and ADE20k) demonstrate the effectiveness of our FedSeg. We hope this work will attract more attention from the FL community to the challenging semantic segmentation federated learning.
BibTeX
@inproceedings{miao2023fedseg,
  author = {Jiaxu Miao and Zongxin Yang and Leilei Fan and Yi Yang},
  title = {{FedSeg: Class-Heterogeneous Federated Learning for Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00777},
}
Figure