Abstract
Deep learning models for semantic segmentation often struggle with domain shift when deployed in new environments. While Unsupervised Domain Adaptation (UDA) typically requires simultaneous access to both source and target data, real-world constraints like privacy and storage often make source data unavailable. This paper introduces $MAS^3$ (Model Adaptation for Source-Free Semantic Segmentation), a novel framework that represents source domain knowledge as a Gaussian Mixture Model (GMM) in an embedding space. By aligning unlabeled target data to these stored prototypes using Sliced Wasserstein Distance, $MAS^3$ enables effective adaptation without source data. Evaluations on synthetic-to-real benchmarks show that $MAS^3$ achieves competitive performance, significantly outperforming source-only baselines.
Introduction
Semantic segmentation is a critical task for applications like autonomous driving, yet models are notoriously sensitive to the "domain gap" between training (source) and deployment (target) environments. For instance, a model trained on synthetic data from video games often fails on real-world city streets. Standard UDA techniques bridge this gap by training on both datasets together. However, "source-free" adaptation is increasingly necessary for privacy-sensitive or resource-constrained scenarios where the original training data cannot be retained.
Methodology: The $MAS^3$ Framework
The proposed $MAS^3$ framework addresses source-free adaptation through a three-stage process centered on feature-level distribution alignment.
- Prototypical Distribution Estimation: Before discarding source data, the model's feature extractor is used to fit a Gaussian Mixture Model (GMM) in an intermediate embedding space. Each component of the GMM represents a semantic class, serving as a compact "memory" of the source domain.
- Pseudo-Dataset Generation: During adaptation, the model generates "pseudo-samples" from the GMM to simulate source features. Simultaneously, it processes unlabeled target images, generating pseudo-labels for pixels where the model's prediction confidence exceeds a threshold $\tau$.
- Indirect Distribution Alignment: The model is updated by minimizing a combination of Cross-Entropy (CE) loss on pseudo-labeled target samples and the Sliced Wasserstein Distance (SWD) between target features and the GMM prototypes. This geometric optimization "pushes" target features into the clusters defined by the source domain.
Experimental Setup
The authors evaluated $MAS^3$ on two standard synthetic-to-real benchmarks:
- Source Domains: GTA5 (synthetic images from Grand Theft Auto V) and SYNTHIA (urban scene simulations).
- Target Domain: Cityscapes (real-world images of German cities).
These benchmarks represent a significant domain shift, testing the model's ability to generalize from simulated graphics to complex real-world environments.
Results and Analysis
$MAS^3$ demonstrates strong performance despite the lack of source data. On the SYNTHIA $\to$ Cityscapes task, it achieved a mean Intersection over Union (mIoU) of 44.7, outperforming several joint-training UDA methods. On GTA5 $\to$ Cityscapes, the mIoU reached 43.9, a massive improvement over the 24.9 mIoU of the non-adapted source-only model.
A key finding is the impact of the confidence threshold $\tau$ on feature representation. As shown in the t-SNE visualizations below, a higher threshold leads to more distinct class clusters in the embedding space, which correlates directly with improved segmentation accuracy.
The use of Sliced Wasserstein Distance (SWD) proved more stable than traditional adversarial training, providing a robust mechanism for aligning the target distribution with the GMM prototypes.
Conclusion and Future Work
$MAS^3$ provides a powerful, privacy-preserving solution for adapting segmentation models to new domains. By shifting the focus from image-level to feature-level alignment via GMMs, the framework achieves high performance with minimal storage overhead. Future work aims to extend this approach to "lifelong learning" scenarios, where models must sequentially adapt to multiple environments without catastrophic forgetting.
Summary updated