Abstract
Semantic segmentation requires a lot of training data, which necessitates costly annotation. There have been many studies on unsupervised domain adaptation (UDA) from one domain to another, e.g., from computer graphics to real images. However, there is still a gap in accuracy between UDA and supervised training on native domain data. It is arguably attributable to class-level misalignment between the source and target domain data. To cope with this, we propose a method that applies adversarial training to align two feature distributions in the target domain. It uses a self-training framework to split the image into two regions (i.e., trusted and untrusted), which form two distributions to align in the feature space. We term this approach cross-region adaptation (CRA) to distinguish from the previous methods of aligning different domain distributions, which we call cross-domain adaptation (CDA). CRA can be applied after any CDA method. Experimental results show that this always improves the accuracy of the combined CDA method, having updated the state-of-the-art.
Introduction
The primary challenge in semantic segmentation is the high cost of pixel-level annotation. Unsupervised Domain Adaptation (UDA) addresses this by leveraging synthetic data (source domain) to train models for real-world applications (target domain). While standard Cross-Domain Adaptation (CDA) methods align global feature distributions between domains, they often suffer from class-level misalignment. As shown in Figure 1, even when domain distributions are aligned, specific classes may still fall on the wrong side of the decision boundary in the target domain because the alignment is too coarse.
Proposed Method: Cross-Region Adaptation (CRA)
To resolve class-level misalignment, we propose Cross-Region Adaptation (CRA), a plug-and-play framework that can be integrated with existing CDA methods. The core innovation is to perform adversarial alignment within the target domain by distinguishing between "trusted" and "untrusted" regions.
The framework consists of three main steps:
- Initial CDA: A baseline model is trained using a standard UDA method (e.g., ProDA or FADA) to establish initial familiarity with the target domain.
- Region Splitting: For each target image, the model computes an entropy map. Regions with low entropy (high confidence) are designated as "Trusted," while high-entropy regions are "Untrusted."
- Adversarial Alignment: CRA treats these regions as distinct sub-domains. It employs a discriminator to align the feature distributions of untrusted regions with those of trusted regions. Simultaneously, self-training is applied using pseudo-labels from the trusted regions to refine the classifier.
To address class imbalance, the entropy threshold is adjusted for rare classes (e.g., by halving the entropy), preventing them from being disproportionately categorized as untrusted and ensuring they act as anchors for alignment.
Experimental Evaluation
The effectiveness of CRA was evaluated on major sim-to-real benchmarks, including GTA5 → Cityscapes, SYNTHIA → Cityscapes, and Synscapes → Cityscapes.
Quantitative Results
CRA demonstrates universal compatibility, providing consistent performance gains across various baseline CDA methods. When integrated with ProDA, CRA achieved a state-of-the-art mIoU of 58.6% on the GTA5 to Cityscapes task. Improvements across different baselines (AdaptSegNet, ADVENT, FADA, IAST, and ProDA) ranged from 1.1 to 2.9 percentage points in mIoU, confirming that intra-domain alignment provides critical class-level refinement.
Qualitative Analysis
Visualizations indicate that CRA effectively reduces noise and "speckle" errors, particularly near object boundaries and within large, uniform regions like roads where baseline models were previously uncertain. By forcing "confused" pixels to align with "confident" ones in the feature space, the model successfully pushes erroneous predictions toward the correct class boundaries.
Conclusion
Cross-Region Adaptation (CRA) offers a robust solution for class-level alignment in unsupervised domain adaptation. By leveraging the internal consistency of the target domain, CRA bridges the performance gap left by global domain alignment. While dependent on a reasonable initial CDA model, its orthogonality and consistent gains make it a valuable addition to the UDA toolkit. Future research may explore more complex definitions of trust beyond simple entropy.
Summary updated