Spatial Attention Pyramid Network for Unsupervised Domain Adaptation

Congcong Li, Dawei Du, Libo Zhang, Longyin Wen, Tiejian Luo, Yanjun Wu, Pengfei Zhu

Introduction

Deep learning models have achieved remarkable success in computer vision, yet they remain notoriously sensitive to domain shift. This phenomenon occurs when a model trained on a specific source dataset (e.g., clear weather images) experiences a significant performance drop when deployed on a different target domain (e.g., foggy images or artistic renderings). While Unsupervised Domain Adaptation (UDA) aims to bridge this gap without requiring labels for the target domain, existing methodologies often encounter two primary hurdles:

  1. Single-mode alignment: Many methods attempt to align the entire global distribution at once. This coarse approach often fails to account for complex scene layouts and the varying scales of objects within an image.
  2. Contextual Neglect: Traditional approaches frequently focus solely on local patches or global image-level features, overlooking the crucial multi-scale context necessary for identifying objects in challenging or noisy environments.

To address these challenges, the Spatial Attention Pyramid Network (SAPNet) introduces a hierarchical framework designed to align feature distributions by focusing on discriminative regions and rich contextual information across multiple scales.

Methodology

SAPNet moves away from simple global alignment in favor of a nuanced, hierarchical approach. The architecture is built upon three main pillars: the Spatial Pyramid Representation, Task-Specific Guided Spatial Attention, and Dynamic Weight Selection.

The architecture of the Spatial Attention Pyramid Network (SAPNet), illustrating the integration of the backbone, the SAP module, and task-specific heads for adversarial alignment.

1. Spatial Pyramid Representation

Instead of relying on a single feature map, SAPNet constructs a pyramid of $N$ pooled feature maps at varying scales. This structure allows the model to capture both holistic image-level structures and fine-grained local textures simultaneously. A key finding in the research was that average pooling consistently outperformed max pooling in this context. Average pooling better captures the overall strength of local patterns—such as edges and gradients—providing a more stable signal for the domain discriminator than the peak intensities captured by max pooling.

2. Task-Specific Guided Spatial Attention

To ensure the model prioritizes the most informative regions (e.g., a vehicle in a low-visibility foggy scene), SAPNet utilizes "guidance" from task-specific heads. For object detection, it leverages Region Proposal Network (RPN) confidence scores; for segmentation, it uses predicted category maps. This guidance generates spatial attention masks that recalibrate the pyramid features, focusing the alignment process on regions most relevant to the primary task.

3. Dynamic Weight Selection (Channel-wise Attention)

Recognizing that not all pyramid levels are equally useful for every image, SAPNet employs a channel-wise attention mechanism. This component dynamically fuses multi-scale features into a single, highly embedded "semantic vector." This vector is then used for adversarial domain alignment, where a domain discriminator is trained to distinguish between source and target features while the generator (SAPNet) learns to make them indistinguishable.

Experimental Evaluation

The authors evaluated SAPNet across a comprehensive suite of benchmarks covering three distinct domain adaptation scenarios:

  • Real-to-Real (Weather/Sensor shift): Cityscapes $\rightarrow$ FoggyCityscapes, KITTI $\leftrightarrow$ Cityscapes.
  • Synthetic-to-Real: Sim10K $\rightarrow$ Cityscapes, GTA5 $\rightarrow$ Cityscapes, SYNTHIA $\rightarrow$ Cityscapes.
  • Real-to-Artistic: PASCAL VOC $\rightarrow$ Clipart/Watercolor.
Visualization of spatial attention masks and weights across different pyramid scales ($k^1$ to $k^7$) for source and target images, demonstrating how the model focuses on different contextual levels.

Key Results

SAPNet achieved state-of-the-art (SOTA) performance across all three major vision tasks:

  • Object Detection: On the Cityscapes $\rightarrow$ FoggyCityscapes benchmark, SAPNet reached 40.9% mAP, representing a 3% improvement over previous SOTA methods.
  • Instance Segmentation: The model significantly outperformed the SCDA method, increasing performance from 31.4% to 39.4% mAP on FoggyCityscapes.
  • Semantic Segmentation: On the GTA5 $\rightarrow$ Cityscapes task, it achieved a Mean Intersection over Union (mIoU) of 43.2%, comparing favorably against specialized models like CLAN.
  • Ablation Studies: Removing the spatial attention pyramid ("w/o SA") resulted in a massive performance drop (e.g., from 42.2% to 35.4% on PASCAL $\rightarrow$ Clipart), confirming that the pyramid structure is the primary driver of the model's success.

Discussion and Conclusion

The success of SAPNet highlights that context is king in domain adaptation. By utilizing a pyramid, the model identifies objects within their broader environmental context (e.g., a boat relative to water and sky) across multiple scales, making feature alignment robust to noise and domain-specific artifacts. Furthermore, using task-specific predictions to gate attention creates a virtuous cycle where the model learns which regions are most "transferable."

Despite its strengths, the framework involves a trade-off between the number of pyramid levels ($N$) and computational overhead. While $N=13$ provided optimal results for detection, it increases parameter count. Future work may explore extending this multi-scale attention approach to other domain-sensitive tasks such as video action recognition or depth estimation.

Summary updated

Abstract

Unsupervised domain adaptation is critical in various computer vision tasks, such as object detection, instance segmentation, and semantic segmentation, which aims to alleviate performance degradation caused by domain-shift. Most of previous methods rely on a single-mode distribution of source and target domains to align them with adversarial learning, leading to inferior results in various scenarios. To that end, in this paper, we design a new spatial attention pyramid network for unsupervised domain adaptation. Specifically, we first build the spatial pyramid representation to capture context information of objects at different scales. Guided by the task-specific information, we combine the dense global structure representation and local texture patterns at each spatial location effectively using the spatial attention mechanism. In this way, the network is enforced to focus on the discriminative regions with context information for domain adaptation. We conduct extensive experiments on various challenging datasets for unsupervised domain adaptation on object detection, instance segmentation, and semantic segmentation, which demonstrates that our method performs favorably against the state-of-the-art methods by a large margin. Our source code is available at https://isrc.iscas.ac.cn/gitlab/research/domain-adaption .
BibTeX
@inproceedings{li2020spatial,
  author = {Congcong Li and Dawei Du and Libo Zhang and Longyin Wen and Tiejian Luo and Yanjun Wu and Pengfei Zhu},
  title = {{Spatial Attention Pyramid Network for Unsupervised Domain Adaptation}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58601-0\_29},
}
Figure