Abstract
Recent advances in deep learning, especially deep convolutional neural networks, have led to great performance improvement over semantic segmentation systems. Unfortunately, training deep neural networks (DNNs) requires a humongous amount of labeled data, which is laborious and costly to collect and annotate. Thus, plenty of works have proposed an alternative solution to ease the training set creation by using synthetic data. However, models trained on these kinds of data usually under-perform on real images due to the well-known issue of domain shift. To address it, we propose a generative adversarial network (GAN)-based framework called category-level adversarial adaptation networks (CAA-Nets) for domain adaptation in the context of semantic segmentation. Considering semantic predictions that contain spatial and structure information of images, our idea is to make use of this character by imposing discriminators on the semantic predictions. Different from existing works, the proposed framework utilizes a category-level discriminator in the output space to shrink the gap between real and synthetic images. Similar to reinforcement learning, we take final results as a guide to update parameters in the right direction by use of the output-based discriminator. Moreover, to further enhance the performance, we construct an image-based generator and discriminator pair to distill the feature representations obtained by a DNN. Taking advantage of these modules, our model can achieve competitive performance compared with some existing methods. To showcase the generality and scalability of our approach, we evaluate the proposed method on the Cityscapes dataset by adapting from GTAV and SYNTHIA datasets, where the results demonstrate the effectiveness of our method.
1. Introduction
Semantic segmentation—the task of assigning a specific category label to every pixel in an image—is a cornerstone of modern computer vision, particularly for autonomous driving systems. However, the success of deep convolutional neural networks (DNNs) in this field is heavily gated by the requirement for massive, high-quality labeled datasets. Creating such data is an expensive and time-consuming endeavor; for example, manually annotating a single image from the Cityscapes dataset can take a human expert up to 90 minutes.
To bypass this bottleneck, researchers often utilize synthetic data from photo-realistic video games like Grand Theft Auto V (GTAV). While these environments provide automated, pixel-perfect labels, models trained on them often suffer from "domain shift." This phenomenon occurs when a model overfits to the specific textures and lighting of the synthetic source domain and fails to generalize to real-world target domains. Unsupervised Domain Adaptation (UDA) aims to solve this by training models on labeled synthetic data and unlabeled real data to learn domain-invariant features.
2. Methodology: The CAA-Net Framework
The proposed Category-Level Adversarial Adaptation Network (CAA-Net) addresses domain shift through a multi-pronged adversarial strategy. Unlike previous methods that attempt global feature alignment, CAA-Net focuses on structural and category-specific consistency.
2.1 Output-Based Multi-Category Discriminator ($D_{out}$)
The core innovation of CAA-Net is the multi-category discriminator. In standard adversarial adaptation, a single discriminator typically evaluates the entire segmentation map. CAA-Net instead "slices" the output predictions into individual category maps. By feeding these category-specific maps into a discriminator, the model is forced to learn the spatial and structural priors inherent to each class (e.g., the typical shape and location of a "car" versus a "sidewalk"). This component acts as a "tutor," providing a supervisory signal that guides the segmentation network to produce outputs that are structurally plausible in both domains, similar to how rewards guide agents in reinforcement learning.
2.2 Image-Based Reconstruction and Distillation
To ensure that the intermediate features learned by the backbone are truly domain-invariant, CAA-Net incorporates a generator-discriminator ($G-D$) pair. The generator ($G$) attempts to reconstruct the original RGB image from the latent features produced by the encoder. If a single generator can successfully reconstruct both synthetic and real images from the same feature space, it implies that the encoder has successfully distilled the essential semantic information while discarding domain-specific noise. A minor segmentation network ($D_{seg}$) is employed alongside the generator to ensure that this reconstruction process preserves the semantic integrity of the scene.
3. Experimental Evaluation
3.1 Datasets and Setup
The effectiveness of CAA-Net was evaluated using two primary synthetic-to-real adaptation benchmarks:
- GTAV → Cityscapes: Adapting from 24,966 synthetic images to 5,000 real-world urban scenes.
- SYNTHIA → Cityscapes: Adapting from 9,400 synthetic images to the same real-world target.
The primary metric used is the mean Intersection over Union (mIoU), which quantifies the accuracy of the segmentation boundaries.
3.2 Results and Discussion
CAA-Net achieved state-of-the-art or highly competitive results across both benchmarks. In the GTAV → Cityscapes task, the model achieved a 30.9% mIoU, outperforming established baselines such as CYC (29.6%) and LSD (30.3%). For the SYNTHIA → Cityscapes task, the model reached 30.4% mIoU, a significant improvement over the non-adapted baseline of 17.4%.
Ablation studies confirmed the necessity of each component. Specifically, removing the category-level discriminator ($D_{out}$) resulted in a performance drop from 30.9% to 27.8% mIoU, highlighting its critical role in enforcing structural consistency across domains.
4. Conclusion
CAA-Net provides a robust framework for unsupervised domain adaptation in semantic segmentation. By combining category-level adversarial learning with feature distillation via image reconstruction, the model effectively bridges the gap between synthetic and real-world data. While the framework shows great promise, challenges remain in accurately segmenting small or infrequent categories, such as traffic signs and motorcycles. Future research will focus on optimizing the network for higher resolutions—addressing the non-linear relationship between image size and accuracy—and improving the representation of these "long-tail" classes without significantly increasing computational overhead.
Summary updated