AutoAdapt: Automated Segmentation Network Search for Unsupervised Domain Adaptation

Xueqing Deng, Yi Zhu, Yuxin Tian, Shawn Newsam

Abstract

Neural network-based semantic segmentation has achieved remarkable results when large amounts of annotated data are available. However, such data is expensive to collect, leading to the development of Unsupervised Domain Adaptation (UDA) methods that adapt models trained on synthetic data to real-world domains. Current approaches often overlook how network architecture impacts transferability. This paper introduces AutoAdapt, a framework that performs neural architecture search (NAS) specifically for UDA. We identify an "optimization gap" in searching architectures without target labels and propose a novel metric using maximum mean discrepancy and regional weighted entropy to bridge it. Experimental results demonstrate that AutoAdapt discovers architectures that significantly enhance the performance of existing adaptation techniques.

The Architecture Gap in Domain Adaptation

In semantic segmentation, the high cost of manual labeling often necessitates training models on labeled synthetic "source" data (e.g., GTA5) and adapting them to unlabeled real-world "target" data (e.g., Cityscapes). While most UDA research focuses on loss functions or data augmentation, the role of the network architecture itself is frequently ignored.

A critical observation in this work is the lack of correlation between supervised performance and UDA performance. As shown in Figure 1, modern architectures like DeepLabV3+ and DAnet outperform the older DeepLabV2 in supervised settings but exhibit significantly lower performance when adapted to a new domain. This suggests that architectural features optimized for a specific dataset may actually hinder generalization.

Comparison of supervised vs. UDA performance across different architectures, highlighting the lack of correlation between the two.

The AutoAdapt Framework

Standard NAS requires a labeled validation set to evaluate candidate architectures. In UDA, the absence of target labels creates an optimization gap. AutoAdapt addresses this through a two-stage automated framework.

The AutoAdapt framework: Stage I involves searching and building child models via evolution, while Stage II focuses on domain adaptation and evaluation using a label-free metric.

Stage I: Search and Build

Using an evolutionary algorithm, AutoAdapt mutates a "seed-net" (a pretrained backbone) to generate child models. The search space encompasses several critical hyperparameters and modules, including crop size, layer count, kernel sizes, dilation rates, and Squeeze-and-Excitation (SE) layers.

Stage II: Adaptation and Evaluation

Child models undergo standard UDA training (e.g., adversarial training). To rank these architectures without target labels, AutoAdapt employs a joint proxy metric:

  1. Maximum Mean Discrepancy (MMD): Measures the distance between source and target feature distributions.
  2. Regional Weighted Entropy (ReEnt): Evaluates prediction consistency. By partitioning images into regions of similar appearance and weighting entropy, this metric rewards models that produce smooth, consistent predictions in visually similar areas.

Key Insights and Results

The framework was evaluated on standard benchmarks: GTA5 → Cityscapes and SYNTHIA → Cityscapes.

  • Superior Performance: On GTA5 → Cityscapes, AutoAdapt achieved a Mean Intersection over Union (mIoU) of 50.2%, surpassing the DeepLabV2 baseline (43.8%) and previous state-of-the-art methods.
  • Architectural Preferences: The search consistently favored Squeeze-and-Excitation (SE) layers, which help align feature distributions by reweighting channel importance. Additionally, optimized dilation rates improved prediction smoothness without increasing computational overhead.
  • Robustness: Architectures found by AutoAdapt proved more transferable than those discovered by standard NAS methods (like DARTS), which tend to overfit the source domain.

Conclusion

AutoAdapt demonstrates that architecture is a first-class citizen in domain adaptation. By bridging the optimization gap with a label-free proxy metric, the framework automates the discovery of robust, transferable networks. Future work will explore expanding the search space and developing more efficient, differentiable NAS methods tailored for unsupervised scenarios.

Summary updated

Abstract

Neural network-based semantic segmentation has achieved remarkable results when large amounts of annotated data are available, that is, in the supervised case. However, such data is expensive to collect and so methods have been developed to adapt models trained on related, often synthetic data for which labels are readily available. Current adaptation approaches do not consider the dependence of the generalization/transferability of these models on network architecture. In this paper, we perform neural architecture search (NAS) to provide architecture-level perspective and analysis for domain adaptation. We identify the optimization gap that exists when searching architectures for unsupervised domain adaptation which makes this NAS problem uniquely difficult. We propose bridging this gap by using maximum mean discrepancy and regional weighted entropy to estimate the accuracy metric. Experimental results on several widely adopted benchmarks show that our proposed AutoAdapt framework indeed discovers architectures that improve the performance of a number of existing adaptation techniques.
BibTeX
@misc{deng2021autoadapt,
  author = {Xueqing Deng and Yi Zhu and Yuxin Tian and Shawn Newsam},
  title = {{AutoAdapt: Automated Segmentation Network Search for Unsupervised Domain Adaptation}},
  year = {2021},
  eprint = {2106.13227},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.13227},
}
Figure