Introduction: The Challenge of Class Heterogeneity
Federated Learning (FL) enables collaborative model training across distributed parties without sharing raw data. While successful in image classification, semantic segmentation introduces a unique obstacle: Class Heterogeneity. In real-world scenarios, different clients (e.g., autonomous vehicles in different cities) may only encounter a subset of object classes. For instance, Client A might see "motorbikes" while Client B sees "pedestrians."
In standard FL, Client A would label pedestrians as "background," forcing its local model to suppress these features. When these conflicting updates are aggregated, the global model suffers from severe "client drift" and poor convergence. Existing weight-level regularization methods are often too coarse to resolve these fine-grained, dense prediction conflicts.
Methodology: The FedSeg Framework
To mitigate these issues, FedSeg introduces two core innovations to the Federated Averaging (FedAvg) pipeline:
1. Modified Cross-Entropy Loss ($L_{backce}$)
Standard cross-entropy ignores background pixels, which in this context are actually "unseen" foreground classes. FedSeg aggregates the probabilities of all classes not present in a client's local dataset and treats the background as a mixture of these missing classes. This provides a negative gradient signal that prevents the local model from incorrectly suppressing features of classes it hasn't seen yet.
2. Local-to-Global Pixel Contrastive Learning ($L_{con}$)
To maintain alignment with the global semantic space, FedSeg uses contrastive learning. Local pixel embeddings are pulled toward the global representation of the same class and pushed away from others. To ensure computational efficiency, the authors use a Pixel-to-Region approach: local pixels are compared against compact "region embeddings" (class anchors) derived from the global model, rather than a dense pixel-to-pixel comparison.
Experimental Evaluation
FedSeg was evaluated on four major benchmarks: Cityscapes, CamVID, PascalVOC, and ADE20k. These datasets were partitioned to simulate extreme class heterogeneity, where clients might only observe 1 or 2 classes.
Key Results
- Significant Accuracy Gains: On the Cityscapes dataset under non-IID conditions, FedSeg achieved a 50.24% mIoU, a massive leap from the 10.40% achieved by standard FedAvg.
- Superior Convergence: FedSeg consistently outperformed state-of-the-art FL algorithms like FedProx, FedDyn, and MOON by 4-5% mIoU across various settings.
- Efficiency: The framework demonstrates faster convergence, often reaching target accuracy in half the communication rounds required by competing methods.
Insights and Discussion
The primary insight of FedSeg is that the "background" in heterogeneous FL is not empty but contains hidden foreground classes. Addressing this at the loss-function level is more effective than architectural changes. Furthermore, using the global model as a "semantic anchor" provides a reliable teacher for local clients.
Limitations
While effective, FedSeg's contrastive component is sensitive to the quality of pseudo-labels generated by the global model. Additionally, while it solves class heterogeneity, future work is needed to address domain heterogeneity (e.g., varying lighting or weather conditions across clients) and open-world scenarios where the class set is not static.
Summary updated