TL;DR
This paper tackles the challenge of adapting semantic segmentation models trained on synthetic urban scenes to perform well on real-world data. They propose a novel "curriculum learning" approach that first learns easier global and local label distributions from target domain data, then uses these to regularize the main segmentation task, significantly improving performance.
What Problem Does This Paper Address?
Semantic segmentation, a crucial task for applications like autonomous driving, requires vast amounts of labeled data. While synthetic data generation offers a way to create large datasets with pixel-perfect annotations, a significant "domain mismatch" exists between synthetic and real images. This mismatch, caused by differences in rendering, lighting, textures, and camera perspectives, severely degrades the performance of models trained on synthetic data when applied to real scenes. Existing domain adaptation methods often struggle with the complex, structured nature of semantic segmentation labels.
Methods
The core of this paper is a curriculum domain adaptation strategy. Instead of directly aligning features between domains, they propose a multi-stage learning process:
- Easy Tasks First: The approach first tackles "easier" tasks that are less sensitive to the domain gap. These include estimating:
- Global Label Distributions: The proportion of pixels belonging to each semantic category within an entire image. This is easier because urban scenes have predictable global characteristics (e.g., roads occupy a large portion).
- Local Label Distributions: Similar distributions but focused on "landmark superpixels" (small, representative regions within an image). This provides more localized guidance.
- Regularized Segmentation: The learned distributions from the target domain are then used as posterior regularization to constrain the training of the main semantic segmentation network. This means the network's predictions on real images are guided to be consistent with these inferred global and local properties, effectively bridging the domain gap without requiring direct feature alignment.
- Color Constancy: As an additional preprocessing step, they incorporate a computational color constancy method to align the color distributions of synthetic and real images, further reducing the domain mismatch.
Datasets
- Source Domains (Synthetic):
- SYNTHIA: A large dataset of synthetic urban scenes.
- GTA: Generated from the Grand Theft Auto V game, offering photo-realistic synthetic urban scenes.
- Target Domain (Real):
- Cityscapes: A widely used, high-quality dataset of real urban scenes captured from vehicles.
Key Insights and Ideas
- Curriculum Learning for Domain Adaptation: The idea of breaking down the complex domain adaptation problem into simpler, sequential learning steps (global/local distributions before pixel-wise segmentation) is a key innovation.
- Leveraging Domain Idiosyncrasies: Urban scenes have inherent structural properties (e.g., road layout, building placement) that are relatively invariant across domains and can be exploited to infer useful label distributions.
- Posterior Regularization: Using inferred target domain properties to regularize the segmentation network's output is a powerful way to guide adaptation without relying on direct feature alignment.
- Complementarity of Tasks: Global and local label distributions offer complementary information. Global distributions correct overall class imbalances, while local distributions provide spatial guidance.
- Color Constancy as a Plug-in: The color calibration step is presented as a general-purpose enhancement that can be applied to various domain adaptation methods.
Key Results
- Significant Performance Gains: The proposed curriculum domain adaptation approach consistently outperforms baseline methods (trained without adaptation) and other domain adaptation techniques on both SYNTHIA-to-Cityscapes and GTA-to-Cityscapes tasks.
- Improved mIoU: For example, on GTA2Cityscapes, their method (Ours (CC+I+SP)) achieves an mIoU of 31.4%, compared to 21.1% for the baseline (NoAdapt).
- Ablation Studies: Extensive experiments demonstrate the effectiveness of individual components, such as the global label distribution (Ours (I)) and landmark superpixels (Ours (SP)), and show that combining them (Ours (I+SP)) yields the best results.
- Color Constancy Benefits: Applying color constancy further boosts performance, with Ours (CC+I+SP) achieving the highest mIoU.
- Qualitative Improvements: Visual results show that their method produces much more accurate and coherent segmentation masks on real images compared to the baseline.
Limitations and Future Work
- Indistinguishable Classes: The paper notes that some classes, like "train" and "bus," are inherently difficult to distinguish even for human annotators due to subtle visual cues and dataset limitations. This suggests a need for better synthetic data or more labeled real data for these specific classes.
- Exploration of More Properties: The authors suggest exploring additional target properties beyond label distributions that are easier to estimate and can be formulated as functions of pixel-wise labels.
- Generalizability: While the approach is shown to be agnostic to the backbone network, further testing with a wider variety of architectures could be beneficial.
- "Market Value" of Synthetic Data: The paper quantifies the benefit of synthetic data, suggesting that synthetic images can significantly reduce the need for labeled real data, especially when real datasets are small. This highlights the economic value of synthetic data in reducing annotation costs.
Summary updated