Source Domain Subset Sampling for Semi-Supervised Domain Adaptation in Semantic Segmentation

Daehan Kim, Minseok Seo, Jinsun Park, Dong‐Geol Choi

TL;DR

More data isn't always better. This paper introduces Source Domain Subset Sampling (SDSS), a method that improves semi-supervised domain adaptation by identifying and training only on the most "meaningful" subset of source data. By filtering out confusing or irrelevant synthetic samples, the authors achieve state-of-the-art performance in semantic segmentation while simultaneously reducing training time.

What Problem Does This Paper Address?

In semantic segmentation, training deep models requires massive amounts of pixel-perfect labels, which are expensive to produce for real-world data. Researchers often use synthetic datasets (like GTA5 or SYNTHIA) as a "source" because labels are free. However, there is a significant domain gap between synthetic and real-world "target" images.

While Semi-Supervised Domain Adaptation (SSDA) uses a small amount of labeled target data to bridge this gap, it typically treats the entire source dataset as equally useful. The authors argue this is a mistake. Synthetic environments often contain "confusing" samples—like a car driving on a sidewalk due to loose simulation constraints—that mislead a model when it tries to generalize to the real world. These irrelevant samples act as noise, hindering the adaptation process.

Methods

The authors propose a three-stage pipeline to distill a high-quality subset from the source data:

  1. Knowledge Acquisition (Stage 1): A segmentation network is pre-trained on a small available set of labeled target images. This biases the network toward the target domain's characteristics.
  2. Pixel-level Sampling (Stage 2): This "target-biased" model predicts labels for the entire source dataset. The method then performs a "sanity check": it only keeps source pixels where the model’s prediction matches the source ground truth. This effectively filters out synthetic features that don't translate well to the target domain.
  3. Image-level Sampling (Stage 3): To select the best full images, the authors introduce a class-balance score ($\tau_C$). This score rewards images that have high prediction correctness but also penalizes images dominated by a single "easy" class (like sky or road). This ensures the final subset is diverse and challenging enough for robust training.

The resulting subset is significantly smaller than the original source but contains higher-density information relevant to the target task.

Datasets

The paper evaluates SDSS on standard benchmarks and introduces a new one:

  • GTA5 $\to$ Cityscapes: Adapting from a video game to real-world German street scenes.
  • SYNTHIA $\to$ Cityscapes: Another synthetic-to-real urban benchmark.
  • Ocean Ship (New): The authors created a maritime dataset consisting of 500 real images and 200,000 synthetic images. They noted a lack of large-scale maritime datasets for domain adaptation, making this a valuable contribution for coastal surveillance research.

Key Insights and Ideas

  • The "Confusing Sample" Hypothesis: The most striking insight is that large-scale synthetic data can be counterproductive. Humans know a car shouldn't be on a sidewalk, but a model trained on a simulation where that happens might learn incorrect geometric priors.
  • Quality Over Quantity: By using only ~70% of the source data, the model often performs better than when using 100%. This challenges the "big data" dogma and suggests that data curation is as important as model architecture.
  • Synergistic Effects: SDSS isn't a standalone model but a data-loading strategy. It can be "plugged in" to existing SSDA frameworks (like MME or ASS), consistently boosting their performance.

Key Results

  • State-of-the-Art Performance: On the GTA5 $\to$ Cityscapes benchmark with 100 labeled target samples, SDSS improved the baseline mIoU (mean Intersection over Union) by 4.27 points.
  • Efficiency: The method achieved higher accuracy with fewer training samples, leading to shorter training cycles.
  • Maritime Success: On the new Ocean Ship dataset, the method provided a massive 9.13 mIoU improvement over the baseline, proving its scalability to new domains.
  • Ablation Success: The authors showed that training on the excluded (rejected) samples led to a substantial drop in performance, validating that their sampling logic successfully identifies "unhelpful" data.

Limitations and Future Work

While highly effective for segmentation, the current pixel-level sampling logic is specific to that task. The authors briefly explored instance-level sampling for object detection (using bounding boxes instead of pixels) and found promising results on the Foggy Cityscapes dataset, but noted this needs more optimization.

Additionally, the method relies on the initial small target dataset being "representative" enough to act as a filter. If the initial target samples are outliers, the sampling process might inadvertently discard useful source data. Future work could involve making the sampling process more iterative or adaptive as the model learns.

Summary updated

Abstract

In this paper, we introduce source domain subset sampling (SDSS) as a new perspective of semi-supervised domain adaptation. We propose domain adaptation by sampling and exploiting only a meaningful subset from source data for training. Our key assumption is that the entire source domain data may contain samples that are unhelpful for the adaptation. Therefore, the domain adaptation can benefit from a subset of source data composed solely of helpful and relevant samples. The proposed method effectively subsamples full source data to generate a small-scale meaningful subset. Therefore, training time is reduced, and performance is improved with our subsampled source data. To further verify the scalability of our method, we construct a new dataset called Ocean Ship, which comprises 500 real and 200K synthetic sample images with ground-truth labels. The SDSS achieved a state-of-the-art performance when applied on GTA5 to Cityscapes and SYNTHIA to Cityscapes public benchmark datasets and a 9.13 mIoU improvement on our Ocean Ship dataset over a baseline model.
BibTeX
@misc{kim2022source,
  author = {Daehan Kim and Minseok Seo and Jinsun Park and Dong‐Geol Choi},
  title = {{Source Domain Subset Sampling for Semi-Supervised Domain Adaptation in Semantic Segmentation}},
  year = {2022},
  eprint = {2205.00312},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.00312},
}
Figure