Towards Adaptive Semantic Segmentation By Progressive Feature Refinement

Bin Zhang, Shengjie Zhao, Rongqing Zhang

TL;DR

This paper introduces a novel "progressive feature refinement" (PFR) framework to improve unsupervised domain adaptation for semantic segmentation. By progressively aligning intermediate feature maps from source and target domains and using adversarial learning on the output, their method significantly boosts performance on unseen target domains without requiring target labels.

What Problem Does This Paper Address?

Semantic segmentation, a crucial computer vision task that assigns a category label to every pixel in an image, often suffers from a "domain shift" problem. Models trained on one dataset (e.g., synthetic images) perform poorly when applied to another with different data distributions (e.g., real-world images). Collecting and annotating large datasets for every new domain is expensive and time-consuming. Unsupervised domain adaptation aims to bridge this gap by leveraging labeled source data to train a model that generalizes well to unlabeled target data. However, existing methods often struggle with high-level vision tasks like semantic segmentation, where complex relationships between objects need to be preserved.

Methods

The proposed approach, termed "progressive feature refinement" (PFR), enhances existing semantic segmentation networks (specifically, DeepLab-v2 with a ResNet-101 backbone) with two key components:

  1. Progressive Feature Refinement (PFR) Module: This module disentangles the "style" and "content" of feature maps at multiple stages of the network. It then aligns these features between source and target domains progressively. The alignment is achieved by minimizing a combined "style loss" (matching Gram matrices of features) and "content loss" (matching feature maps directly). This staged alignment aims to make the features more domain-invariant.
  2. Domain Adversarial Learning (Output Space): To further ensure domain-agnostic outputs, an auxiliary domain classifier is introduced. This classifier is trained to distinguish whether a segmentation output comes from the source or target domain. The segmentation network is then trained to "fool" this classifier, encouraging it to produce outputs that are indistinguishable between domains.

The overall framework is trained end-to-end by combining the standard segmentation loss, the PFR loss, and the adversarial loss.

Datasets

  • Source Domain: GTA5 (24,966 high-definition synthetic images with automatic annotations).
  • Target Domain: Cityscapes (2,975 training, 1,525 testing, 500 validation samples of real-world urban scenes).

The GTA5 dataset is used as the source domain due to its rich annotations, and the Cityscapes dataset serves as the target domain for evaluating cross-domain generalization.

Key Insights and Ideas

  • Progressive Alignment is Key: Instead of aligning features only at a single stage, aligning them progressively across multiple network layers helps to disentangle domain-specific styles from essential semantic content more effectively.
  • Style vs. Content Disentanglement: Separating and aligning style and content features is a powerful mechanism for domain adaptation, allowing the model to retain semantic understanding while shedding domain-specific visual characteristics.
  • Output-Level Adversarial Training: Applying adversarial learning not just to intermediate features but also to the final segmentation output provides an additional strong signal for domain invariance, directly improving the quality of the predictions on the target domain.
  • Memory Efficiency: Compared to methods that use image-to-image translation, the PFR approach is more memory-efficient as it avoids generating intermediate translated images.

Key Results

The proposed method achieves state-of-the-art performance on the GTA5 → Cityscapes unsupervised domain adaptation task. Specifically, it reaches the best mean Intersection over Union (mIoU) compared to other baseline approaches. The paper highlights significant improvements in categories like "road," "building," "wall," "fence," "terrain," "rider," "truck," and "bus." Qualitative results show more precise object boundary segmentation and smoother outputs on target domain images.

Limitations and Future Work

  • Performance on Less Frequent Objects: The paper acknowledges that PFR might have limited performance gains on some less frequent object classes, which could potentially lead to negative transfer.
  • Future Work: While not explicitly detailed, the success on this challenging task suggests that extending PFR to other domain adaptation scenarios or more complex segmentation tasks could be promising. Further exploration into optimizing the trade-off parameters (λadv, λpfr) for different datasets and tasks might also be beneficial.

Summary updated

Abstract

As one of the fundamental tasks in computer vision, semantic segmentation plays an important role in real world applications. Although numerous deep learning models have made notable progress on several mainstream datasets with the rapid development of convolutional networks, they still encounter various challenges in practical scenarios. Unsupervised adaptive semantic segmentation aims to obtain a robust classifier trained with source domain data, which is able to maintain stable performance when deployed to a target domain with different data distribution. In this paper, we propose an innovative progressive feature refinement framework, along with domain adversarial learning to boost the transferability of segmentation networks. Specifically, we firstly align the multi-stage intermediate feature maps of source and target domain images, and then a domain classifier is adopted to discriminate the segmentation output. As a result, the segmentation models trained with source domain images can be transferred to a target domain without significant performance degradation. Experimental results verify the efficiency of our proposed method compared with state-of-the-art methods.
BibTeX
@inproceedings{zhang2020towards,
  author = {Bin Zhang and Shengjie Zhao and Rongqing Zhang},
  title = {{Towards Adaptive Semantic Segmentation By Progressive Feature Refinement}},
  booktitle = {ICIP},
  year = {2020},
  url = {https://openreview.net/forum?id=TJJamIuT8D},
}
Figure