Spatial Attention Pyramid Network for Unsupervised Domain Adaptation

Congcong Li, Dawei Du, Libo Zhang, Longyin Wen, Tiejian Luo, Yanjun Wu, Pengfei Zhu

TL;DR

This paper introduces SAPNet, a novel spatial attention pyramid network designed to tackle the challenge of unsupervised domain adaptation in computer vision tasks like object detection, instance segmentation, and semantic segmentation. By effectively leveraging multi-scale contextual information through a spatial pyramid and a task-specific guided attention mechanism, SAPNet significantly improves performance on datasets with domain shifts.

What Problem Does This Paper Address?

Deep learning models excel in computer vision tasks, but their performance often degrades when applied to data from a different domain than they were trained on (domain shift). This is a major hurdle for real-world applications where data distributions can vary significantly (e.g., clear weather vs. foggy weather, synthetic vs. real images). Unsupervised domain adaptation (UDA) aims to bridge this gap by training models on labeled source data and unlabeled target data, without requiring any labels from the target domain. Existing UDA methods often struggle by globally aligning domain distributions or by not effectively capturing multi-scale contextual information, leading to suboptimal performance, especially in complex scenarios.

Methods

The core of SAPNet lies in its Spatial Attention Pyramid Network (SAPNet) architecture, which combines two key innovations:

  • Spatial Pyramid Representation: Inspired by spatial pyramid pooling, SAPNet constructs a multi-scale representation of feature maps. This is achieved by applying average pooling with different kernel sizes to the backbone network's feature maps. Each pooled feature map captures semantic information at a different scale, allowing the network to understand objects and scene layouts at various levels of detail. Unlike traditional SPP, SAPNet's pyramid is designed to capture context across all levels, not just a few fixed windows.
  • Task-Specific Guided Spatial Attention Mechanism: To make the spatial pyramid more effective, SAPNet employs a guided spatial attention mechanism. This mechanism uses task-specific information (e.g., object confidence maps from a region proposal network for detection, or semantic category predictions for segmentation) to guide the attention process. This allows the network to focus on the most discriminative semantic regions and feature maps within the pyramid, effectively learning to adapt to the target domain by highlighting relevant contextual cues. The attention masks are learned and normalized using a softmax function, recalibrating spatial responses adaptively.

The overall optimization involves minimizing both a task-specific loss (for the primary vision task) and an adversarial loss to align the source and target domain distributions.

Datasets

The paper evaluates SAPNet on several challenging domain adaptation benchmarks across object detection, instance segmentation, and semantic segmentation:

  • Object Detection:
    • Cityscapes to Foggy Cityscapes
    • Cityscapes to KITTI
    • KITTI to Cityscapes
    • PASCAL VOC to Clipart
    • PASCAL VOC to Watercolor
    • Sim10K to Cityscapes
  • Instance Segmentation:
    • Cityscapes to Foggy Cityscapes
  • Semantic Segmentation:
    • GTA5 to Cityscapes
    • SYNTHIA to Cityscapes

Key Insights and Ideas

  • Multi-scale context is crucial for domain adaptation: Simply aligning global features isn't enough. SAPNet's spatial pyramid effectively captures hierarchical contextual information, which is vital for understanding objects and scenes across different domains.
  • Task-specific guidance enhances attention: Using information directly related to the task (like object proposals or semantic predictions) to guide the attention mechanism significantly improves the model's ability to focus on relevant features for adaptation. This is more effective than generic attention.
  • Average pooling is surprisingly effective: The paper finds that average pooling, rather than max pooling, for the spatial pyramid representation yields better results. This suggests that capturing the overall strength of local patterns (like edges and textures) is more beneficial for domain adaptation than simply selecting the most prominent features.
  • Attention pyramid is more than just multi-scale features: The proposed attention pyramid mechanism, which adaptively weights features from different scales based on task-specific guidance, is shown to be more effective than simply using multi-scale features or global attention.

Key Results

SAPNet demonstrates state-of-the-art performance across various UDA tasks:

  • Object Detection: Achieves significant improvements, for example, a 3% mAP gain on Cityscapes to Foggy Cityscapes. It also shows substantial gains on other datasets, outperforming previous methods by a large margin in some categories.
  • Instance Segmentation: Outperforms existing methods like SCDA by a significant margin (39.4% vs. 31.4% mAP) on the Cityscapes to Foggy Cityscapes task.
  • Semantic Segmentation: Achieves comparable accuracy to state-of-the-art methods on GTA5 to Cityscapes and SYNTHIA to Cityscapes.

Limitations and Future Work

  • Hyperparameter Tuning: The paper mentions that the hyper-parameter $\lambda$ (controlling the trade-off between task-specific and adversarial loss) needs to be tuned for different tasks and domain similarities.
  • Computational Complexity: While not explicitly stated as a limitation, the use of a spatial pyramid with many levels (N=13) suggests potential computational overhead, though the paper focuses on demonstrating effectiveness.
  • Future Work: The paper's ablation studies suggest that increasing the number of pyramid levels generally improves performance, indicating that exploring even deeper pyramids or more sophisticated multi-scale fusion strategies could be a future direction. The effectiveness of task-specific guidance also points towards further research in how to best extract and utilize this information for adaptation.

Summary updated

Abstract

Unsupervised domain adaptation is critical in various computer vision tasks, such as object detection, instance segmentation, and semantic segmentation, which aims to alleviate performance degradation caused by domain-shift. Most of previous methods rely on a single-mode distribution of source and target domains to align them with adversarial learning, leading to inferior results in various scenarios. To that end, in this paper, we design a new spatial attention pyramid network for unsupervised domain adaptation. Specifically, we first build the spatial pyramid representation to capture context information of objects at different scales. Guided by the task-specific information, we combine the dense global structure representation and local texture patterns at each spatial location effectively using the spatial attention mechanism. In this way, the network is enforced to focus on the discriminative regions with context information for domain adaption. We conduct extensive experiments on various challenging datasets for unsupervised domain adaptation on object detection, instance segmentation, and semantic segmentation, which demonstrates that our method performs favorably against the state-of-the-art methods by a large margin. Our source code is available at https://isrc.iscas.ac.cn/gitlab/research/domain-adaption.
BibTeX
@inproceedings{li2020spatial,
  author = {Congcong Li and Dawei Du and Libo Zhang and Longyin Wen and Tiejian Luo and Yanjun Wu and Pengfei Zhu},
  title = {{Spatial Attention Pyramid Network for Unsupervised Domain Adaptation}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58601-0\_29},
}
Figure