SCMix: Stochastic Compound Mixing for Open Compound Domain Adaptation in Semantic Segmentation

Kai Yao, Zhaorui Tan, Zixian Su, Xi Yang, Jie Sun, Kaizhu Huang

Abstract

Semantic segmentation models for autonomous driving often struggle when transitioning from synthetic training data to diverse, real-world conditions. Open Compound Domain Adaptation (OCDA) addresses this by requiring models to adapt to multiple unlabeled target domains while generalizing to entirely unseen "open" environments. We present SCMix, a novel data augmentation strategy that stochastically mixes a single source image with multiple target domain images using a grid-based approach. SCMix significantly improves robustness by embracing intra-domain variance, achieving state-of-the-art performance on benchmarks like GTA5 to C-Driving and demonstrating superior zero-shot generalization to unseen datasets.

Introduction

In autonomous driving, the "domain gap" is a critical hurdle. A model trained on synthetic, sunny data (the Source) often fails when encountering real-world rain, snow, or fog (the Target). While standard Unsupervised Domain Adaptation (UDA) typically treats the target as a single homogeneous domain, real-world environments are "compound"—a mixture of various subdomains.

Open Compound Domain Adaptation (OCDA) is a more realistic and challenging setting. It tasks models with adapting to a known set of unlabeled target domains (e.g., Rainy, Snowy, Cloudy) while maintaining the ability to generalize to "open" domains (e.g., Overcast) never seen during training. Previous OCDA approaches often relied on "divide-and-conquer" strategies, separating target data into distinct subdomains. However, this ignores the complex relationships between these conditions, limiting the model's robustness in truly novel scenes.

Conceptual overview of SCMix and its impact on the generalization span in Open Compound Domain Adaptation.

The SCMix Methodology

The core innovation of this work is Stochastic Compound Mixing (SCMix). Unlike standard UDA methods that perform one-to-one mixing (one source image with one target image), SCMix leverages the diversity of the entire compound target domain during each training step.

Multi-Target Mixing and Grid Masking

Instead of selecting a single target image, SCMix stochastically samples $N_c$ images from the compound target set. These images are fused using a dynamic Grid Mask ($M_{grid}$). The image space is divided into a grid, and each cell is randomly populated by a patch from one of the $N_c$ sampled target images. This creates a "compound-mixed" target image—a mosaic that captures the variance across different weather conditions and environments simultaneously.

Class-Mix Integration

To maintain semantic consistency, SCMix incorporates Class-Mix. Specific object classes (e.g., cars, pedestrians, signs) are extracted from the labeled source image using a class mixing mask ($M_{class}$) and pasted onto the compound-mixed target mosaic. This allows the model to learn object features against a highly varied and representative background, forcing it to focus on domain-invariant characteristics.

The SCMix pipeline: combining multiple target images via grid masking and integrating source class information.

Transformer Backbone

SCMix is the first OCDA method to utilize the SegFormer architecture (specifically MiT-B5). The authors find that the self-attention mechanism in transformers is inherently more robust to domain shifts than traditional CNN-based architectures like DeepLabv2, providing a stronger baseline for adaptation tasks.

Theoretical Foundation

The authors provide a theoretical justification for SCMix using group theory to derive a new generalization bound for OCDA. They prove that SCMix is a generalized extension of previous mixing methods. By better approximating the joint distribution of all target subdomains, SCMix theoretically yields a lower empirical risk, effectively minimizing the "adaptation gap" between the source and the diverse target/open domains.

Experimental Evaluation

The researchers evaluated SCMix using synthetic source domains (GTA5, SYNTHIA) and the C-Driving dataset for compound targets (Rainy, Snowy, Cloudy). Open-set generalization was tested on Overcast images from C-Driving, as well as the Cityscapes, KITTI, and WildDash datasets.

Key Results

SCMix achieved state-of-the-art (SoTA) performance across all benchmarks:

  • GTA5 → C-Driving: SCMix reached a mean Intersection over Union (mIoU) of 46.1%, a 5.9% improvement over the previous best method, ML-BPM.
  • Open Set Generalization: On unseen datasets like Cityscapes and KITTI, SCMix outperformed existing methods by a wide margin, achieving 50.1% mIoU on the GTA5 → OpenSet task (vs. 40.8% for the previous SoTA).
  • Robustness on Hard Classes: Significant gains were observed in difficult categories such as "Rider" and "Bus," which typically suffer in domain adaptation.

Conclusion

SCMix demonstrates that intra-domain variance is a powerful tool for building robust semantic segmentation models. By embracing the complexity of compound domains through stochastic mixing and grid masking, SCMix forces models to learn features that are invariant across seen conditions, naturally aiding in zero-shot generalization to unseen environments. While the computational cost of large transformer backbones is a consideration, the substantial gains in mIoU and robustness make SCMix a significant step forward for safe autonomous navigation in unpredictable real-world settings.

Summary updated

BibTeX
@article{yao2025scmix,
  author = {Kai Yao and Zhaorui Tan and Zixian Su and Xi Yang and Jie Sun and Kaizhu Huang},
  title = {{SCMix: Stochastic Compound Mixing for Open Compound Domain Adaptation in Semantic Segmentation}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2025},
  doi = {10.1109/tnnls.2025.3560318},
}
Figure